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ABSTRACT 


A new approach was recently proposed to effectively and 
objectively evaluate risk management methodologies and tools 
for their suitability to a given organizational situation. 
The proposed approach, known as CERTS, is based on defining 
suitability in terms of criteria which in turn are described 
in terms of attributes and metrics. Using the Analytic 
Hierarchy Process, this thesis develops the CERTS approach 
into a Decision Support System, that could be used easily and 
effectively by organizations for selecting a risk management 
methodology or tool. The thesis also applies the developed 
DSS to three case studies to gain insights on _ the 


applicability of the DSS. 
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ime INTRODUCTION 


A. BACKGROUND 

The need for acceptable computer security risk management 
practices is becoming more evident throughout the federal and 
commercial environment because of the sophistication and 
complexity of today's technology and the increased value 
society has placed on information. Research over the last 
four to five years has focused on establishment and refinement 
of a formalized framework for risk management (Katzke, 1988 
and Mayerfeld, 1989), and many automated tools have been 
developed by commercial and governmental organizations. 
Despite the attention given to the development of a framework, 
little has been done to establish a technique for determining 
which risk management methodology or tool is most suitable for 
a given organizational situation. 

To overcome this deficiency, a new method was recently 
proposed to effectively and objectively evaluate risk 
management methodologies and tools for their suitability to a 
given organizational situation (Garrabrants, Ellis, Hoffman, 
and Kamel, 1990). The proposed approach, known as CERTS, is 
based on defining suitability in terms of criteria which in 
turn are described in terms of attributes. These attributes 


are further decomposed into metrics that could objectively be 


applied to the methodology or tool under consideration for a 
given organizational situation. A mathematical model could 
then be used to combine metric evaluations with weights 
assigned to criteria, attributes and metrics to obtain an 
overall suitability index of each alternative methodology or 
tool. 

By using the proposed methodology, determining the 
suitability of particular method or tool becomes standardized, 
flexible, and expandable. The method is standardized since a 
uniform set of criteria, attributes, and metrics are used. 
The method is also flexible because different weights could be 
assigned to metrics, attributes, and criteria according to the 
organizational situation. Finally, as the definition of 
suitability is refined, the method is expandable by simply 


adding additional metrics, attributes, and criteria. 


B. OBJECTIVE 

The objective of this research is to develop the CERTS 
method into a Decision Support System (DSS) that could be used 
easily and effectively by management personnel for selecting 
a risk management methodology or tool. Currently the proposed 
method relies on a series of manual questionnaires which are 
tedious and time consuming. 

In addition to building a Decision Support System the 
research aims at applying the proposed method. To accomplish 


this goal, we apply the developed DSS to three hypothetical 


case studies. Each case study represents a different 
environment that requires the use of a risk management tool to 
assess the risks that each environment faces. 

Based on the application of the method to the case 
studies, we expect to gain useful insight that could be used 
later to refine the method, by adding, removing, or modifying 
criteria, attributes or metrics, to accurately select the most 
appropriate methodology or tool to fit the particular 


organizational requirement. 


C. RESEARCH QUESTION 

The main research question addressed by this thesis is: 
Can an effective Decision Support System based on the CERTS 
approach be developed to assist organizations in selecting the 
most appropriate risk management tool for their environment? 

A secondary research question is to determine whether the 
developed decision support system could be applied 
successfully in different environments to select the best risk 


management tool. 


D. SCOPE, LIMITATIONS AND ASSUMPTIONS 

For the purpose of this study, the decision support system 
was limited to three risk management packages. The three 
packages selected for inclusion in this study were based upon 
recommendations by the National Institute of Standards and 


Technology (NIST) Risk Management Laboratory. 


The case studies developed in Chapter V are hypothetical 
Situations developed from cases used by NIST for testing and 
evaluating the risk management packages at the laboratory. 
These situations have been expounded upon by the authors to 
actually test the CERTS Decision Support System. 

To allow for testing, comparisons, and familiarization of 
the risk management packages, the authors spent three days at 
the NIST Risk Management Laboratory in Gaithersburg, Maryland. 
Nicki Lynch and Irene Gilbert were invaluable in helping the 


authors evaluate the three selected packages. 


E. RESEARCH METHODOLOGY 

To accomplish our objective, the methodology consists of 
three phases: 1) Literature Review, 2) Implementing the 
Comparative Method, and 3) Testing the Proposed Method by 
applying it to three case studies. These phases are detailed 
below. 

1. Literature Review 

First, Garrabrants and Ellis" thesis (Garrabrants and 

Ellis, 1990) was reviewed for background information on CERTS. 
Second, Thomas Saaty's Analytical Hierarchy Process (Saaty, 
1980) was examined. Three candidate risk management tools 
were selected, tested, analyzed, and compared at the National 
Institute of Standards and Technologies Risk Management 


Laboratory for inclusion in the developed DSS. 


2. Implementing the Comparative Method 

We have found the Expert Choice software to be an 
excellent vehicle for implementing the proposed technique into 
a Decision Support System. Expert Choice implements the 
Analytic Hierarchy Process (AHP), an approach to multi- 
criteria decision making problems (Saaty, 1982). Under this 
approach, a decision problem is structured in the form of a 
hierarchy (tree). The root of the tree is the goal. 
Intermediate levels of the tree represent the criteria used to 
accomplish the goal, and at the bottom of the tree are the 
leaves which represent the alternative choices. Users make 
comparative judgements in order to establish the relative 
importance between criteria and the preference of the 
alternatives with respect to the specific qualities of a 
criterion. 

CERTS fits nicely within the framework of AHP. 
Concepts of criteria, attributes, and metrics could be 
incorporated readily at the intermediate levels of an AHP 
decision hierarchy. At the bottom of the tree would be the 
candidate methodologies or tools under consideration. Since 
the proposed metrics are boolean questions, they need to be 
modified and expressed ina formthat allows the assignment of 
numeric rather than boolean values. 

The proposed Decision Support System served as the 
structure for integrating the suggested modifications to the 


boolean questions. The system assigned numeric weights to the 


modified CERTS method for each methodology or tool. This 
process completed the development of the CERTS Decision 
Support System. 
3. Testing the Proposed Method 

In this phase, the developed Decision Support System 
is tested by applying it to three case studies. The case 
studies were developed via input from NIST and the authors. 
Information inferred from the case studies was applied to the 
prototype Decision Support System to make a recommended 


selection for each case situation. 


F. ORGANIZATION OF STUDY 

Chapter II reviews the CERTS approach. Chapter III 
explains the underlying premise of the Analytic Hierarchy 
Process used as the vehicle for implementing the decision 
Support system. Chapter IV describes the implementation of 
the CERTS Decision Support System. Chapter V details the 
application of the decision support system to three case 
Scenarios and discusses the results of the DSS for each case. 
Chapter VI gives concluSions and recommendations about the 


research and indicates directions for further research. 


II. CERTS: A COMPARATIVE EVALUATION METHOD FOR RISK 
MANAGEMENT METHODOLOGIES AND TOOLS 
A. INTRODUCTION 
This chapter is designed to assist the reader in 

understanding the basics of the Comparative Evaluation Method 
for Risk Management Methodologies and Tools (CERTS). CERTS is 
an evaluation method that uses metrics to determine the 
suitability of a risk management methodology or methodological 
tool for a particular organizational situation. It was 
developed by Major William M. Garrabrants and Major Alfred W. 
Ellis III both from the Computer Technology Curriculum, Naval 
Postgraduate School (Garrabrants and Ellis, 1990). The 
motivation behind their work is to develop a methodology for 
comparing the large number of risk management methodologies 
and tools available today. These methodologies and tools were 
developed largely as a result of the decentralization of 
automated data processing (ADP) systems and the increased 
breadth of the information stored in the systems. As 
Professor Lance Hoffman noted in the 1986 National Computer 
Security Conference: 

One significant lack today is metrics for risk analysis 

and risk management. There is no currently accepted set 

of criteria against which all methods can be compared. It 


is difficult to evaluate or to convey the advantages and 
disadvantages of a given methodology or tool when no 


accepted evaluation metric exists. (Hoffman, 1986, p. 
S77) 
With the development of CERTS, an effort has been directed 
toward the establishment of metrics for the evaluation of risk 
analysis and risk management methods and the appraisal of the 


numerous automated risk management tools currently available. 


B. THE CERTS APPROACH 

As stated above, the major objective for developing CERTS 
was to develop a new technique to effectively and objectively 
evaluate available risk management methodologies and tools 
for organizations and to establish a means of comparing these 
methodologies and tools. Garrabrants and Ellis concluded, 
through their preliminary research, that risk analysis 
criteria are a vital component of the selection of any risk 
management procedure. Their research lead them to believe 
that metrics could provide the means to measure a tool or 
package for suitability, thus assisting the user in selecting 
the most appropriate methodology for a given situation. This 
belief solidified their ultimate objective in establishing a 
standard set of metrics that could be used to evaluate risk 
management methodologies and tools for an organizational 
Situation. 

During the initial approach to this study, they discovered 
there was no existing technique to compare the risk management 


methodologies. Therefore, they developed an example of a 


model, a paradigm, that promoted the comparison of risk 
management methods utilizing factors such as suitability, 
quality or acceptability. The ultimate purpose of this 
approach was to remove the analysts’ deficiencies or biases 
from the evaluation, thereby assisting the analyst in 
determining which methodology should be selected. 
1. A MeasSure for Suitability 

Technology has brought an abundance of new risks that 
must be understood and addressed within the risk management 
arena. Businesses, companies, federal agencies, andall users 
of computer technology must be able to plan and forecast for 
the probability of adverse events. Numerous quantitative and 
analytical methods for risk management and decision-making 
under uncertainty have been developed, but the question still 
remains, "Which method is best for a particular situation?" 

At this point the authors established a list of 
prerequisites a risk manager must possess in order to 
successfully accomplish this task. This list addressed the 
necessity of understanding the system being managed, its 
suitability to the purpose of the organization, and a thorough 
understanding of a majority of the methods available. Several 
risk management methods were found to be available for 
determining risks. Among those reviewed were Quantitative, 


Checklist, Scenario, Questionnaire methodologies, and hybrids 


of each. The results revealed that each method has its own 
strengths and weaknesses that depend on the nature of its use. 
2. Steps for Suitability 

Garrabrants and Ellis concluded in their literature 
research that a great deal of effort had gone into the 
development of risk management methodologies, but that the 
methodologies lacked criteria and standardization. The 
application and development of their criteria for evaluation 
of computer security risk management methodologies followed 
those of Merkhofer (Merkhofer, 1987), but differs in the 
introduction of metrics which reduce the subjectivity of the 
criteria. 

The next guestion to be addressed is how suitability 
would be defined. Suitability is defined as _ those 
characteristics of a risk management methodology or tool that 
are pertinent and appropriate for the requirements of a 
particular person, organization, system, and/or situation 
(Garrabrants and Ellis, 1990). The steps to measuring 
Suitability is summarized in Table l. 

By implementing these _ steps, the analysis’ of 
Suitability became standardized, flexible, and expandable. 
All criteria could now be compared consistently across all 
methods and could provide the user with the capability of 


expanding and weighing the criteria to meet his requirements. 
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This process resulted in the culmination of seven 


criteria composed of between two and four attributes. The 
criteria are: consistency, useability, adaptability, 
feasibility, completeness, validity, and credibility. (See 
Table 2.) 


TABLE 1. STEPS FOR MEASURING SUITABILITY 


Establish a set of criteria that describes a 
method's suitability. 


Define the suitability criteria in terms of related 
attributes. 


Specify metrics that describe the presence of the 
attributes. 


Make a quantitative statement of the appearance of 
the suitability criteria by determining the ratio of 
actual occurrences of the metric to the number of 
possible occurrences. 


Use the derived quantitative values for each of the 
criteria to evaluate and compare the variety of 
methods and tools available to the organization. 





C. CRITERIA, ATTRIBUTES, AND METRICS 

Once the seven criteria were developed, the authors 
selected the unweighted normative relationship model to 
formulate a simple mathematical relationship between the 
metrics and their associated criteria. The derived 
measurements of each attribute were viewed as a set, applied 
to a mathematical expression in boolean terms, and expressed 
as a ratio. In turn, each attribute within a criteria was 


summed to determine the ratio for that criteria. After 


il 


determining each criteria'’s ratio, the ratios were summed and 
applied to a mathematical expression resulting in a 


Suitability index ratio. 


TABLE 2. SUITABILITY CRITERIA 





| Consistency. Given a particular system configuration, 
results obtained from independent analysis will not 
i significantly differ. 






: Useability. The effort necessary to learn, operate, 
prepare input, and interpret output is generally worth 
the results obtained. 











Adaptability. The structure of the method or tool can be 
applied to a variety of computer system configurations 
(and the inputs can be easily updated as they 

| periodically change). 










| Feasibility. The required data is available and can be | 
| economically gathered. 








| Completeness. Consideration of all relevant 
| relationships and elements of risk management is given. 







| ' 
| Validity. The results of the process represent the real | 
| phenomenon. | 


| 
| Credibility. The output is believable and has merit. 


Throughout the process of developing the criteria, their 






associated attributes, and metrics, the authors came to the 
conclusion that not all of the criteria could be maximized 
Simultaneously. Some criteria are maximized at the expense of 
others. Thus, determining the best risk management tool or 
method would require trading one desirable trait for another. 
Therefore, the suitability of a method could be determined 
only after integrating the needs of an organization with the 


process as developed in this thesis. 


eZ 


1. Application of the Metrics 

Now that a means of evaluating the suitability of risk 
management methodologies existed through the utilization of 
metrics, the method was augmented. To gain an appreciation of 
their validity, Garrabrants and Ellis applied their metrics to 
four sample, intuitively understandable methods of risk 
analysis. The four methods included: Annual Loss Expectancy 
(ALE), checklist, scenario, and questionnaire. 

Using this approach, intuitive predictions were made 
for each of the criteria. The purpose of analyzing their 
results in the context of their predictions was to provide an 
approximation of the usefulness and integrity of the metrics. 
In essence, this process confirmed the metrics evaluation 
technique by providing an acceptable, standardized measurement 
of a methodology's attributes upon which to base a more 
sophisticated comparison of risk management tools. 

The significance of the metric evaluation is in its 
application to hybrid methodologies. Hybrid methodologies are 
representative of the majority of tools that are currently 
available to computer security risk managers. The strength of 
the metrics evaluation technique was demonstrated by 
evaluating and comparing a small sample of four hybrid tools. 

2. Performance of the Metrics 
The evaluation results were focused on three different 


perspectives. These perspectives consisted of examining the 
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results of each tool separately, examining the results of each 
tool in comparison to each other, and finally, examining the 


results by comparing the suitability index of each tool. 


D. CONCLUSION 

Garrabrants and Ellis established a standardized set of 
metrics in a structured relationship that may be used to 
evaluate risk management methodologies and tools for their 
Suitability in a given organizational situation. The metrics 
were successfully applied to four computer security risk 
management methodologies to develop an informal validation. 
The metrics were also used to evaluate four hybrid computer 
security risk management tools as a test and demonstration of 
the multiple criteria evaluation method. Its versatility was 
exemplified by the successful application to dissimilar tools. 
Several suggestions for extension of the concepts developed in 


their research were provided to guide future research. 
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III. THE ANALYTIC HIERARCHY PROCESS 


A. INTRODUCTION 

The Analytic Hierarchy Process (AHP) is a theory for 
modeling unstructured problems in the economic, social, and 
management sciences. AHP was developed by Thomas L. Saaty of 
the Wharton School, University of Pennsylvania (Saaty, 1980). 
AHP models a decision process as a hierarchy or a system of 
stratified layers, with the top layer being the ultimate goal 
or decision that needs to be made, and each succeeding layer 
being the criteria, subcriteria, subsubcriteria, etc. of the 
hierarchy. Finally, the leaf nodes represent the alternatives 
of the decision process. A pairwise comparison is made on 
each level of the decision tree to determine the importance of 
criteria and subcriteria, as well as the preference of the 
alternatives with respect to these criteria. 

AHP is designed to consider as many relevant facts and 
ideas as possible to assist managers who have difficult 
decisions. When making these difficult decisions, managers 
normally consider the two or three major elements of a complex 
decision. Quite often other elements which play an integral 
role in the decision process may not be able to be considered. 


The AHP process helps to alleviate this oversight. 


Jus) 


Palrwise comparison in AHP is more advantageous than the 
process of assigning weights. When assigning weights, all 
criteria are considered together with the most important 
criteria assigned the highest weight. This weighting process 
is used in assigning weights for all the succeeding ranked 
Criteria. In pairwise comparison, each criterion is compared 
against each and every criterion to determine which criteria 
is most important of the two and by how much. The AHP process 
automatically calculates the weights for each criteria. 

Once the hierarchy is established it may easily be 
modified. The manager does not have to start from scratch. 
New branches may be added to the hierarchy, and the 
comparisons remade. If a branch in the hierarchy attains a 
higher level of importance, the pairwise comparisons may be 
reevaluated with a bigger weight assigned by the process to 


that branch. 


B. THE AHP PROCESS 

The first step in setting up the AHP is to construct the 
hierarchy. Hierarchies are developed by the decision maker by 
establishing the necessary criteria to be considered. The 
hierarchy may be established from the top down or the bottom 
up. When a level becomes too complex or may not be readily 
compared, the element of that level may be broken down into 
newer lower levels, with finer distinctions. Even after this 


hierarchical development, modifications may be made by adding 
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new nodes (e.g., criteria, subcriteria, alternatives) to the 
decision process model. 

The top level of a hierarchy is called the focus or the 
broad overall objective. The second layer represents the 
major criteria used in making the decision. Subsequent layers 
are subcriteria that further explain the major criteria. The 
leaves or bottom nodes of the tree are the alternatives from 
which the decision maker wants to select in order to 
accomplish the objective of the decision problem. Each layer 
may have numerous elements, although Saaty states that five to 
nine iS an appropriate amount. 

Figure 1 is a simple example of a hierarchy. The focus or 
the overall objective of the hierarchy is to select the best 
job. The second layer consists of the criteria used in making 
a decision. For this example, they include wage, location, 
and potential. The third and final layer in this example 
represents the alternatives available to the decision maker. 
In this example, they are IBM, APPLE, and NCR. Changes may 
readily be made to the hierarchy by adding new alternatives, 
such as Compaq, or by adding, deleting, or changing factors to 
be used in the determination of the job to be selected. For 
example, benefits could be substituted for potential in the 
hierarchy. The process of selecting a job for decision making 
may require a complex hierarchy. Every possible element 


relevant to the selection process should be included in the 


Ly 


hierarchy in order to allow the best possible decision to be 


made. 


SELECT A JOB 
WAGE LOCATION POTENTIAL 


Figure 1. Select a Job Hierarchy 


The second step in AHP consists of establishing or setting 
priorities among the elements of the hierarchy. The setting 
of priorities is established by pairwise comparisons within 
each layer of the hierarchy. Each comparison in the hierarchy 
is assigned a number from one to nine, with one being the 
items are of equal importance and nine being the one element 
having absolute importance over the other. The judgement of 
the ranking and importance of the items is at the discretion 
of the individual performing the comparison. Other values are 
dispersed between those two extremes. Pairwise comparisons 
may also be made using number or language (letters or verbal) 
assignments, depending on the preference of the user. Terms 
such as weakly more important, strongly more important, or 


absolutely more important may assist in the development of 
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complex pairwise comparisons. Table 3 shows the complete 
breakdown of the pairwise comparison criteria for AHP. 

Comparisons of elements within a hierarchy may be made by 
placing the results into a matrix. The matrix format is based 
upon the number of elements in the hierarchy. The matrix of 
the example of Figure l is a three by three matrix, as shown 
in Table 4. An important question when making pairwise 
comparisons 1s: 

How much more strongly does this element ... posses - or 
contribute to, dominate, influence, satisfy, or benefit - 
the property than does the element with which it is being 
compared? (Saaty, 1982, page 77) 

The individual uses his judgment, knowledge, or his 
awareness of the situation to assign these values. The first 
comparison in Table 4 is made between wage and location. In 
this particular example, wage is assumed to be weakly more 
important than location, and therefore, a value of three was 
assigned, as indicated in Table 3. The reverse of this 
comparison, i.e. location to wage, has a reciprocal value of 
the wage to location comparison weight, or 1/3. The element 
in the left hand column of the matrix is always compared to 
the element in the top row of the matrix. The intensities are 
determined by the decision maker, through pairwise comparison, 
judgment, knowledge, or his particular awareness of a given 
Situation. Wage has been demonstrated to be strongly favored 


to slightly dominant when compared to potential. Thus a 


Ly 


TABLE 3. THE PAIRWISE COMPARISON SCALE (Saaty, Decision 
Making for Leaders age 78 


ae me re ee ere ee nee ee ne ee oS a 


Intensity of | Definition ae —— 
_Importance — 


—— SSS Se eS me 


Eouare importance of | Two elements 
both elements contribute equally to 
the property 


Weak importance of Experience and 
both element over judgment slightly 


another favor one element 
over another 


Essential or strong | Experience and 
importance of one judgment strongly 
element over favor one element 
another over another 












An element is 
strongly favored and 
its dominance is 
demonstrated in 
practice 


Demonstrate 
importance of one 
element over 

another 














The evidence favoring 
one element over 
another is of the 
highest possible 
order of affirmation 





Absolute importance 
of one element over 
another 






















Compromise is needed 
between two judgments 


Intermediate values 
between two 
adjacent judgments 









If activity i has 
one of the 
preceding numbers 
assigned to it when 
compared with 
activity j, then j 
has the reciprocal 
value when compared 


Reciprocals 
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figure of six is assigned, and the reciprocal potential to 
wage is assigned a 1l/6. The complete matrix is shown in Table 


4. 


TABLE 4. SELECT A JOB COMPARISONS 


[See eee eee ee potential 


SS ES ERE a a 


ee EY 


rept Gn. mall 






| Potential © 





The next step, termed synthesis (Saaty, 1982) is to set 


the overall priorities for a decision problem. Synthesis is 
the pulling together of all the values and arriving at one 
number to indicate the priority of that element. Table 5 
illustrates this step in the synthesis of results of Select a 
Job Model. The columns of the matrix are totaled, and each 
entry in the column is then divided by the total of that 
column to obtain a normalized matrix, as shown in Table 6. 
This process allows comparison among the elements. 

The average of each row is then computed by taking the sum 
of each row and dividing this sum by the number of entries in 
that row, as shown in Table 7. This gives the percentage of 
the overall priority for each element. 

In this particular example, the wage criterion is the 
element which will have the largest impact on the decision on 


which job to take, as it is has the highest value. 
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An important item to consider is the consistency of the 
matrix derived through pairwise comparisons. An inconsistency 
could be introduced if, for example, an individual prefers the 
IBM job over the Apple job, the Apple job over the NCR job, 
but the NCR job over the IBM job. The overall consistency of 
the pairwise comparison matrix can be computed by means of an 
inconsistency ratio. The inconsistency ratio does not need to 
be exactly zero. If the value obtained is under 10%, then the 
pairwise comparison matrix is considered to be consistent. If 
the ratio is over 10%, then the pairwise comparisons are 


considered to be inconsistent and should be reevaluated. 


TABLE 5. SYNTHESIS OF SELECT A JOB 


Ce ee a Potential 
jecrtis yas Pa 





TABLE 6. NORMALIZED MATRIX OF SELECT A JOB 


= =< 
Potential ae a ae 
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TABLE 7. OVERALL PRIORITIES FOR SELECT A JOB 


[Element | wage _| tocation | potential 
a ea =1.96/3 = 
=0.75/3 = .25 


| Potential _— Sr =0.29/3 = .10 


Consider the scenario shown in Table 8. In this scenario, 









wage is weakly more important than location and potential. 
Location is weakly more important than potential. The 
percentage of overall relative priorities is determined and 


presented in Table 9. 


TABLE 8. MATRIX FOR INCONSISTENCY RATIO CALCULATION 


Soe Seer Potential 


a a a 


| Potential __ ie, Sy a Te 


Column 1.66 
| Totals— 


To determine if an inconsistency has been introduced into 
















the decision process, each column value is multiplied by the 
relative priority for that criterion, i.e., the wage column 
with the wage priority of .57. The entries in each row are 
then totaled as shown in Table 10. Each row sum is divided by 


its corresponding relative priority as shown in Table ll. 
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TABLE 9. PRIORITIES FOR INCONSISTENCY RATIO CALCULATION 


Element Location }] Potential Average 
Row Sums 
1 S 3 Pe Te 1.72 = 
57 
Location 1/3 l 3 0.86 0.86/3 = 
.29 
.14 


TABLE 10. INCONSISTENCY RATIO CALCULATIONS 


Element (.57) (.29) (.14) Row Total 
Wage Location Potential 
fwage | 57 od ee | ee | eg 
[Location | .19 | .29 | 42 | 90 


potential | _.10 =| 49 __|}- #43: =| oe 













TABLE 11. INCONSISTENCY RATIO CALCULATIONS 


| Wage 186 divided by 0.57 = 3.26 
0.90 divided by 0.29 = 3.10 
0.43 divided by 0.14 = 3.07 


rr rr i i oe re we re ee eee me ee er en a re ee re a ee ee ee ee eS 







The results of this division are summed then divided by 
the number of elements in the matrix to obtain the average. 
From this average the number of elements are subtracted and 
the result is divided by two. This is called the consistency 
index (CI) (Saaty, 1982). The CI in this example is 0.07. 


The inconsistency ratio is obtained by dividing the CI by an 
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average consistency, based on the number of criteria in the 
matrixes. The average consistency value for a matrix of three 
criteria is 0.58 (Saaty, 1982). The inconsistency ratio for 
Select a Jobis .12 or 12%, which is above 10%, indicating an 
inconsistency in the pairwise comparisons. The pairwise 


comparisons should be reevaluated. 


C. CONCLUSION 

By using AHP, an individual may consider many more 
elements than is usually possible in the normal human decision 
thought process. An individual thought process can generally 
consider two to three factors, but with AHP, any number of 
factors can be considered. Even trivial elements which could 
have an impact upon the decision maker may be considered. 
Using a pairwise comparison, more accurate weights are 
calculated for the criteria, resulting in a more refined 


decision. 
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IV. IMPLEMENTATION OF CERTS USING THE ANALYTIC HIERARCHY 
PROCESS 
A. INTRODUCTION 

The purpose of this chapter is to adapt the CERTS method 
to the AHP process, and to develop a decision support system 
(DSS) to assist organizations in the selection of a risk 
management methodology or tool to suit their needs. 

The CERTS technique is useful to an organization in the 
selection of a risk management package. However, this 
technique is hard to apply in its present form. Users must 
analyze a large number of questionnaires, then perform the 
necessary computations manually to determine the best 
available package. CERTS does not have the ability to 
differentiate strengths and weaknesses of certain metrics, as 
it makes boolean determinations only. Weights may not be 
assigned to these criteria to further refine the solution to 
address the priority needs of the organization. The 
application of CERTS is also tedious and time consuming for 
the user. CERTS application requires that the user become 
thoroughly familiar with each risk management package being 
analyzed. 

The AHP process, however, assists in overcoming these 
problems. The process is completely automated, decreasing the 


amount of time required to fill out questionnaires with the 
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calculations being done automatically. The pairwise 
comparisons of AHP allow the assignment of weights’ to 
criteria, attributes and metrics. The user will not have to 
become intimately familiar with each package as weights could 
be assigned to each package in the leaf nodes. 

The DSS selected to incorporate CERTS into AHP was Expert 
Choice, developed by Expert Choice, Incorporated of 
Pittsburgh, Pennsylvania. Expert Choice offers the capability 
of a hierarchy up to six levels deep, with up to seven 
subnodes for each node of the hierarchy. Pairwise comparisons 
may be made at each level. Expert Choice can therefore 


support a decision process with thousands of input criteria. 


B. IMPLEMENTATION OF CERTS USING AHP 

The CERTS methodology is readily adaptable for 
implementation using AHP. The concepts of criteria, 
attribute, and metric in CERTS map nicely into the concepts of 
Criteria, subcriteria, and subsubcriteria in AHP. This is 
explained in the following paragraphs. 

The objective of selecting the best risk management 
package becomes the top layer or goal of the AHP hierarchy. 
The CERTS criteria level becomes the second layer of the 
hierarchy. These are the main decision elements of the DSS 
and are shown in Figure 2. The third level of the hierarchy 
contains the attributes that are used to refine the criteria. 


These attributes correspond to the subcriteria of the AHP 
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method. The fourth level of the hierarchy contains the 
metrics that are used to further define the attributes. These 
metrics expressed as questions in CERTS were modified and 
expressed as subsubcriteria in the AHP hierarchy. These 
subsubcriteria could be used in pairwise comparison, and, 
therefore, assigned weights. For example, the boolean metric 
for the subcriteria reliability, "Does the process provide a 
mechanism to reduce the introduction of personal bias?" is 
transformed into the subsubcriteria of "reducing the 
introduction of personal bias." Then "reducing the 
introduction of personal bias" may be compared with other 
subsubcriteria and assigned a weight. Each criteria is 
discussed in detail in the sections below. Finally, the leaf 
nodes of the hierarchy contains the alternative risk 
management tools from which the most appropriate package will 
be selected. Incorporating the alternatives in the hierarchy 


is explained in Section C. 


Selection of Best Risk Management Package 


Adaptability Completeness Credibility 
User Interface Feasibility Validity 


Figure 2. Criteria of Risk Management Package Hierarchy 
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1. Consistency 

Consistency relates to the ability to duplicate the 
results consistently throughout the process. Consistency has 
subcriteria of reliability and consistent terminology. 
Reliability is concerned with a package's objectivity or the 
reduction of subjectivity in the risk management process. The 
subcriteria of consistent terminology relates to the ability 
of the package to use the same terminology throughout the 
entire risk management - program. The subcriteria and 
subsubcriteria for consistency are listed in Template 1 of 
Appendix A. 

2. User Interface 

User interface is the ability and knowledge needed by 
the user to understand the complete system, as well as the 
level of support provided by the vendor of the system. The 
criteria of user interface is broken down into subcriteria of 
error handling, simplicity, ease of use, understandability, 
and support. Error handling is concerned with the ability of 
the program to identify input errors. Simplicity deals with 
the outward appearance of the package, e.g., does it appear 
easy for the user to understand the process. The ease of use 
subcriteria measures how well structured and logically 
sequential the process is. Understandability relates to the 
ability of comprehending the underlying premise that supports 


the package methodology. Support is concerned with the 
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assistance provided by the program vendor. The subcriteria 
and subsubcriteria for user interface are listed in Template 
2 of Appendix A. 
3. Adaptability 

Adaptability relates to the ability to apply the 
method to various types of computer systems, and whether it 
may be easily updated. Computer systems run the gamut from 
personal computer to mainframe computer to a complex 
distributed network. Adaptability has the subcriteria of 
portability and modifiability. Portability is concerned with 
the ability to use the product across various computer systems 
and configurations. Modifiability is the ability to apply 
different alternatives or options to the process to determine 
the effect upon the outputs. The subcriteria and 
subsubcriteria for adaptability are detailed in Template 3 of 
Appendix A. 

4. Feasibility 

Feasibility is concerned with the cost and amount of 
effort required by the organization to fulfill the information 
requirements and input for the risk management package. 
Subcriteria for feasibility are availability, practicality and 
scope. Availability subcriteria distinguishes between 
internal and external data needed by the system, and the ease 
by which that data may be obtained. Concern with the 


economics of gathering the required data is covered by the 
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subcriteria of practicality. Scope deals with the broadness 
of the system to cover all necessary items contained in the 
organization's information. Template 4 of Appendix A presents 
the subcriteria and subsubcriteria for feasibility. 
5. Completeness 

Completeness 1S concerned with the coverage of all 
risk management areas of concern to the satisfaction of the 
user. Scope, elements, and element attributes are the 
subcriteria for completeness. Scope, which is duplicated in 
other criteria, 1s concerned here with the level of detailed 
analysis that is done throughout the various aspects of the 
organization. Elements deal with the components that operate 
to determine the risks of a= system. Subcriteria of 
completeness are concerned with the outcomes or consequences 
that could occur from the elements) attributes. The 
subcriteria and subsubcriteria of the completeness subcriteria 
are shown in Template 5 of Appendix A. 

6. Validity 

The validity criteria measures the package's ability 
to represent reality of desired legitimate situations. The 
subcriteria for validity are relevancy, scope, and 
practicality. Relevancy means that results of the process are 
meaningful to the organization. Scope is used in the context 
of validity of the process on all the various aspects of the 


organization. Practicality is repeated again from the 
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feasibility criteria, but deals with the validity of the data 
gathered by the process. Template 6 of Appendix A displays 
the criteria and subsubcriteria of the validity criteria. 
7. Credibility 

The last criteria, credibility, deals with whether the 
conclusions arrived at by the package are acceptable by the 
organization. The subcriteria of credibility are 
intuitiveness and reliability. Intuitiveness shows whether 
the results will instill and maintain the confidence of the 
user organization. The ability to obtain repeatable results 
from the package determines the reliability. Template 7 of 
Appendix A exhibits the subcriteria and subsubcriteria for the 
credibility criteria. 

Appendix B shows the output from Expert Choice 


implementing CERTS. 


C. INCORPORATING ALTERNATIVES TO THE HIERARCHY 
l. Alternative Risk Management Packages 

The alternative risk management packages were selected 
in conjunction with inputs from the Department of Commerce's 
National Institute of Standards and Technologies (NIST). The 
risk management packages for this study were selected from 
data obtained from the sampling and extensive testing of 
numerous risk management packages at NIST's Risk Management 
Laboratory. The packages that were selected were LAVA, 


developed by Los Alamos National Laboratory, Los Alamos, New 
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Mexico; BDSS, developed by Ozier, Perry and Associates, San 
Francisco, California; and RISKWATCH, developed by Expert 
Systems Software, Incorporated, Long Beach, California. These 
packages became the leaf nodes of the AHP hierarchy. 

The subsubcriteria in the lower level of the Decision 
Support System, (listed in Templates 1 through 7 of Appendix 
A) were applied to the three packages. The ability of each 
risk management package to meet the subsubcriteria was 
measured by the authors and NIST personnel's qualitative 
opinions. 

2. AsSigning Weights to the Alternatives 

Using pairwise comparison, each risk management 
package was asSigned a weight that indicates its preference 
with respect to each subsubcriterila. If two packages were 
deemed equal in ability by the authors, then the DSS assigned 
equal weights to these packages. For example, it was found 
that in Template 1 of Appendix A, the subsubcriteria 
"establishing standard language" of the subcriteria consistent 
terminology, of the consistency criteria, was addressed 
equally by all three packages (LAVA, BDSS, and RISKWATCH). 
Therefore, pairwise comparisons assigned equal weights to each 
package. This is shown in Table 12. 

Pairwise comparisons were made for all subsubcriteria 
for the DSS. As described in Chapter II, Table 1, comparisons 


may be made by numerical, or verbal methods. In addition, DSS 
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offers a graphical means of presenting the pairwise comparison 
in the form of a pie graph. An example of weights resulting 
from a pairwise comparison is shown in Table 13 for the 


criteria "consistency," the subcriteria "reliability," and for 
the subsubcriteria “reducing the introduction of personal 
bias." Verbal comparisons were made of the alternative risk 
management packages such that LAVA was deemed to be moderately 
more important than BDSS and equal to moderately more 
important than RISKWATCH, while RISKWATCH was deemed to be 
equal to moderately more important than BDSS. Appendix C 


shows all pairwise comparison results for the alternatives in 


regard to the subsubcriterla. 


TABLE 12. EQUALITY IN PAIRWISE COMPARISONS 


Criteria: Consistency 
Subcriteria: Consistent Terminology 
Subsubcriteria: Establishing Standard Language 
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The inconsistency ratio is automatically calculated by 
the DSS for each set of assigned weights. The inconsistency 
ratios were under 10% for all pairwise comparisons of the risk 
management packages. The comparisons were thus deemed to be 


free of inconsistencies. 
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In the next chapter we apply the developed DSS to 


three hypothetical case studies. 


TABLE 13. WEIGHTED ALTERNATIVES SCORES 


Criteria: Consistency 
Subcriteria: Reliability 
Subsubcriteria: Reduces the Introduction of Personal Bias 
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Vie APPLICATION OF THE DSS TO CASE STUDIES 


A. INTRODUCTION 
In Chapter IV, the CERTS methodology was applied to the 
AHP method to develop the CERTS Decision Support System (DSS). 
This DSS may then be used by an analyst to determine the best 
risk management package for a particular computer system site 
or situation. This chapter demonstrates the application of 
the DSS to three different hypothetical case scenarios. These 
case studies were provided by NIST and further developed by 
the authors. 
1. Application of CERTS DSS 

The choice of a suitable risk management package 
depends upon the experience of the analyst and how well he 
tailors the organizational requirements to the evaluation. 
The CERTS DSS could be an invaluable tool in assisting the 
analyst in determining the best package to use. For the 
purpose of this thesis, CERTS DSS includes three risk 
management packages. Additional packages could be 
incorporated easily in the AHP hierarchy using the approach 
detailed in the previous chapter. 

The procedure to apply the CERTS DSS for each case is 
simple, systematic, and straightforward. Initially, the 


analyst conducts pairwise comparisons of the seven criteria at 
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the first level of the DSS, according to the organization's 
particular needs. Consequently, the system assigns weights to 
the various criteria. The analyst may refine the selection 
process by further conducting pairwise comparisons of the 
subcriteria and subsubcriteria of each criteria in the 
hierarchy. 

Upon the completion of each level of pairwise 
comparisons, the system calculates an inconsistency factor. 
If the factor is over 10%, then some type of inconsistency 
exists. The pairwise comparisons should then be reviewed and 
reconsidered until the inconsistency ratio is below 10%. Once 
the weights have been assigned, the synthesis is conducted to 
derive the overall results. The program calculates an overall 
weight for each risk management package based on the pairwise 
comparisons made by the analysts. The program with the 
highest weight is, therefore, the most suitable for the 
organizational situation. 

2. Disclaimer for Case Scenarios 

The case scenarios presented are modeled after test 
cases provided by the Risk Management Laboratory at the 
National Institute of Standards and Technology. The 
information provided by the cases should not be construed to 
represent actual circumstances, conditions, or procedures of 
any kind that may exist in any actual site. The cases were 


developed and designed to provide as realistic and consistent 
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input as possible to the CERTS DSS for the evaluation of the 


risk management packages. 


B. CASE SCENARIO DESCRIPTIONS 
1. Case Scenario One: Distributed Wide Area Network 

(WAN ) 

System X is a nationwide distributed office automation 
and work/project tracking system. The system provides word 
processing, electronic mail, spreadsheets, databases, and 
graphics. In addition to performing its network functions, 
the database serves as a management information system. 

This information system provides management with 
computer listings of the daily and overall functions of each 
office. All work projects are tracked on the database. 
Tracking 1s required for the allocation and purchase of 
resources. The workload is primarily in the format of word 
processing documents. Databases and spreadsheets are used to 
support this function. 

a. Physical Environment 

The system is distributed over nine sites. The 
headquarters (HQ) is located in a Northeastern city, with 
other sites spread around the U.S. at field centers (FCs). At 
HQ, the system is linked via two leased lines to the mainframe 
complex. Each of the FCs' computers is linked into its center 
LAN. All the sites are connected via a network that runs on 


the agency's telecommunications system using a public packet 
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switching service. Backup service at the HQ minicomputer is 
provided for the dial-up access. Access to the HQ mainframe 
is through a packet-switched network to the HQ minicomputer. 
The minicomputer functions as a file and print server for the 
office. 
b. Equipment 
The dollar value of equipment is as follows: 
200 micros @ $3,000 $600,000 


8 small minicomputers @ $75,000 600,000 
1 medium minicomputer @ $200,000 200,000 


25 laptops @ $2,000 50,000 

misc. printers, modems, etc. 100 ,000 

TOTAL $1,550,000 
Equipment used but not owned include: (by 


contractor) packet-switched network; leased lines between HQ 
mini and mainframe computers; and internal networks of various 
types at the different sites. The communications equipment is 
five years old. 
c. Personnel 

All personnel receive critical-sensitive background 
checks before employment. A few administrative personnel 
receive national agency checks (NAC). The management has no 
policy on separation of duties. 

There iS no computer security training. However, 
workers are informed of their physical security 
responsibilities, which include: displaying their picture 


badge at all times; challenging any person not wearing a badge 
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for whose activity or presence appears questionable; reporting 
the loss or misuse of a badge; and surrendering a badge when 
it is no longer needed. A computer security person has been 
assigned by management to track this function. 

There are 300 people at HQ. Each of the FCs have 
between 100 and 150 people. Resource protection measures at 
all sites include: fraud, waste, and abuse education of 
personnel; marking of all equipment; maintaining an active 
inventory of all hardware and software; and making personnel 
responsible for protection of government property. There are 
attractive features (e.g., full color printing) in the systen, 
but no games are allowed. Staff working outside normal hours 
are unsupervised. 

d. Data Environment 

One database is run on the HQ mainframe computer 
and several are run on the HQ minicomputer. Access to the HQ 
mainframe computer is accomplished via a packet-switched 
network, which allows transmission from the HQ minicomputer to 
the mainframe computer over two leased lines. Backups are 
made nightly of the HQ minicomputer and mainframe computer; 
these tapes are stored off-site on a weekly basis. Backups of 
the PCs are made by individual staff members. 

The data is highly sensitive. Accuracy and 
timeliness of the data is required for monthly and semiannual 


reporting. Inaccurate data would result in poor planning and 
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mismanagement of resources. Some of the data requires 
stringent confidentiality protection due to privacy laws. 
Disclosure of this data would result in mission failure, 
dollar loss to the agency, possible lawsuits, and 
embarrassment to the organization. The disclosure, however, 
would not seriously affect the agency mission. Losses for 
disclosure could be $500,000 to $1 million, excluding the cost 
of lawsuits. Losses for mismanagement could be quite costly. 
e. Operating Systems 

The system cannot be described as ‘hacker 
friendly'; there is a warning screen when Signing onto the HQ 
mini- and mainframe computers. The communications equipment 
has not been specially adapted for any site. Remote site 
dial-up users accessing the system receive full processing 
capability. It is not easy to ‘crash’ the applications 
software and break into the operating system or other 
applications. On the other hand, untested software from 
vendors for trial processing is often allowed. This is a 
potential for vulnerability, since no virus detection software 
is available on the system. 

The mini- and mainframe computers have access 
control with passwords, which allow for three tries before 
locking the user ID. Passwords are required to be changed 
every 90 days. Passwords on minicomputers are four characters 


long; passwords on mainframe computer are six to eight 
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characters long. Passwords for both the mini- and mainframe 
computers are: suppressed automatically during entry; 
intentionally related to the user's identity, history, or 
environment; replaced with a new password when forgotten; and 
generated by the user. Management policy prohibits the use of 
group passwords. Passwords and user IDs are removed from the 
minicomputers promptly when an employee leaves. There is no 
timeout on unused accounts. 

Loss of availability of the system for short 
periods (one day) is not a major problem. The loss of the 
large minicomputer for a day, the loss of network, the 
mainframe computer, or the small minicomputer for more than 
seven days would significantly affect productivity. This time 
loss could result in missed mandated monthly and semiannual 
deadlines. Approximate loss of productivity is $500,000 per 
week. The loss of the mainframe computer for a week or more 
or the loss of the network at a critical reporting time would 
result in failure to meet legislated or administrative 
deadlines. While this would produce no dollar loss, goodwill 
would be lost and future budget considerations would suffer. 
The loss of the FCs or HQ for more than a week would be 
disastrous to the agency. The monetary cost would be the 
equipment cost plus loss of productivity at $200,000 per week 
per site. 

Audit and variance detection are implemented. The 


audit trail is read often and handled in a timely manner. A 
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security person checks all unsuccessful logins and system 
bugs. Although technical controls consist of 
authorization/access control, audit trail mechanisms, an 
encryption package, error checking/correcting protocols, and 
user ID and authentication, there is no form of message 
authentication code (MACing) on this system. 

One case of deliberate misuse of resources by 
authorized staff last year was detected at one of the FCs. 
The average level of staff experience with the system is more 
than two years. The turnover in staff averages 15% per year. 
The approximate number of non-staff personnel (e.g., visitors, 
contractors, maintenance) entering the headquarters’ or 
supporting facilities each day is 50. 

f. Management Philosophy and Concerns 

Top management, along with selected members of a 
risk management assessment team, convened to determine their 
major concerns in the selection of a risk management tool with 
the intention of using the CERTS DSS. The committee used the 
pairwise comparison of the DSS to establish their priorities 
for the criteria. Table 14 shows the rankings and summarizes 
the weights assigned to the criteria by the DSS. As Table 14 
indicates, user interface was deemed to be the most important 
criteria in selecting a risk management package. This was 
followed by adaptability, consistency, credibility, validity, 


feasibility, and completeness, respectively. 
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Since System X is a nationwide network and requires 
that each site apply the risk management application, user 
interface is a top priority consideration. With nine sites 
spread across the United States, importance is stressed on 
ease of use, comprehension, and developer support. The 


distance between sites has generated risk management concerns 


TABLE 14. DISTRIBUTED WAN 
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in the areas of input preparation, execution of the process, 
and the interpretation of output. These concerns represent 
the interface and relationships between the analyst and the 
process. The users of the risk package are not required to 
comprehend all features of the process, but do need to 
understand what decisions are expected of them. A process 
that is well structured and logically sequential is critical 
to the ease-of-use aspect. Developer support must include 


complete and extensive documentation, 24 hour phone support, 
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and comprehensive on-site training. Table 15 provides the 
ranking and the DSS assigned weights for the user interface 
subcriterila. 

Given this particular system _ configuration, 
adaptability is high on the list of management concerns. The 
search is for a package or tool that may be applied to a 
variety of computer system configurations. Portability is of 
utmost importance when dealing with a highly distributed 
environment, such as presented in this case. The package must 
apply to a changing environment, as the possibility of adding 
or deleting field sites is high. This change may trigger a 
need to modify the tool to assist the analyst in examining 
alternatives or options. Table 16 summarizes the rankings and 


the DSS assigned weights for the adaptability subcriteria. 


TABLE 15. USER INTERFACE 


i ae 
—— 
a ee 


sf error Handling 


Standardization for risk management is required 








across the entire network. Therefore, the results obtained 
from the risk management package for each site should not be 


significantly different. Consistency implies the ability to 
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TABLE 16. ADAPTABILITY 
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duplicate the results of the process. A key component of 











consistency is reliability. Reliability reduces the wide 
amount of variance that could occur as a result of personal 
biases. The more the process reduces biases in the analysis 
at each site, the more consistent the results will be between 
the analysis teams at each site. Table 17 depicts the 
rankings and weights assigned by the DSS for the consistency 


subcriteria. 


TABLE 17. CONSISTENCY 


DSS_Assigned Weight _ 
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The data used over the WAN is highly sensitive. 













Losses for disclosure could run up to one million dollars. 
Consequently, the credibility aspect of the package is 
essential to the merit of the output. The reliability of the 
risk management package is also essential to its credibility. 


With the possibility of high monetary losses, the same results 
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must occur when the same data is used on different occasions. 
Table 18 shows the ranking for the credibility subcriteria. 
The credibility of a risk management package is 
closely followed by the validity aspect of that package. 
Management wants to avoid the possibility of obtaining 
irrelevant conclusions or results. These results must be 
meaningful to the system. The process should also provide 
categories of solutions rather than specific recommendations. 


Table 19 presents the ranking for the validity subcriteria. 


TABLE 18. CREDIBILITY 
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TABLE 19. VALIDITY 
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The feasibility of obtaining the data is less 
important as each site does its own application. Completeness 
was also aS a minor concern of risk management in this case. 


Therefore, the subcriteria within each of these criteria were 
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considered of equal importance. Tables 20 and 21 display the 
rankings and the weights assigned by the DSS for _ the 
feasibility and completeness subcriteria, respectively. 

The CERTS DSS selected RISKWATCH as the best risk 
management package for the Distributed Wide Area Network (WAN) 
Scenario. The detailed results are shown in Appendix D, 


Templates 1 through 3. 


TABLE 20. FEASIBILITY 
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TABLE 21. COMPLETENESS 
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One of the major advantages of the CERTS DSS is 







that you need not discard the whole framework if you find that 
you overlooked something in formulating the priorities of the 
Criteria. The system is designed to show the sensitivity of 
each criteria to the alternatives. For example, if management 


desired to place more emphasis on consistency, for the above 
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case, the CERTS DSS would select LAVA as the best risk 
management tool for the Distributed Wide Area Network (WAN). 
The sensitivity analysis is illustrated in Appendix D, 
Template 4. 
2. Case Scenario Two: Under Development System - Biomed 
The Biomed system is a new system, currently being 
developed, that is designed to track biomedical research, 
including animal research. Applications will be developed to 
track and record results of experiments and will be used to 
write proposals and reports. The software will include 
relational and hierarchial database packages, word processing, 
and graphics packages. These packages will share data when 
creating reports and presentations. 
a. Physical Environment 
The Biomed system is currently under development 


and will be located in a single tenant government building in 


suburban Washington. The building has no fence and is 
accessible from the street. Site access is controlled by 
Picture ID badges and 24 hour-a-day guards. Visitors with 


proper identification are allowed unescorted into. the 
facility. 

The site has a staff of 1,700. The Biomed system 
will be used by 100 local and 50 remote users. Approximately 


75 non-staff£ personnel (e.g., visitors, contractors, 
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maintenance) enter the site each day. The turnover in staff 
averages 8% per year. 

Each lab and office has a sprinkler system, which 
is part of the building system. All labs have hand-held fire 
extinguishers, but offices do not. All lab personnel have 
been trained in the extinguishers’ use. No smoking is allowed 
in the building. Food and drink are discouraged, but not 
prohibited. Inflammable materials (e.g., solvents) are stored 
and processed at the site. Three fires have occurred in the 
labs within the last two years. 

The Biomed system will be in an existing computer 
room with raised flooring, environmental control, heat 
detectors, drains, and fire suppression. The room is in the 
basement with no windows. Once a month, the floor beneath the 
raised floor is cleaned by a special crew. 

b. Equipment 

Based upon the functional needs and expected usage, 
a minicomputer or small mainframe computer will be procured. 
The expected value of minicomputer and operating system is 
$150,000. Total cost of the application software is estimated 
at $700,000. Existing PCs will be used to access the system. 

c. Personnel 

The agency provides national agency checks (NACs) 

for all employees. There is no computer security training. 


However, personnel are aware of their physical security 
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responsibilities, including: displaying the badge at all 
times; reporting the loss or misuse of ae badge; and 
surrendering a badge no longer needed. Responsibility for 
computer security of the Biomed system will be assigned by 
management. Staff working outside normal hours’ are 
unsupervised. 
d. Data Environment 
The data requires strong integrity protection to 
ensure that published experiment results are _ correct. 
Availability is required for maximizing productivity. Brief 
down times will be inconvenient but not critical. The data is 
time-sensitive and is not made public until experiments and 
analysis are complete to avoid improper interpretation of 
results. 
e. Operating Systems 
On-site Biomed system users will access it through 
a LAN. The proposed method for remote users is through dial- 
in ports; 5 ports are anticipated. There are no dial-up 
communication lines now in place. The communications 
equipment will not be specially adapted. 
f. Administration 
The Animal Rights groups are an active threat. 
These groups have demonstrated at the site, and it is presumed 
they have skilled computer operators within the group. They 


have conducted raids against the site, destroying property and 
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releasing animals. City police are used during 
demonstrations. Forced entry into the building may be 
accomplished, however, forced access into the internal offices 
and labs is difficult. 

As the Biomed system is under development, there 
are no operational or technical controls currently in place. 
There has been no emergency, backup, or contingency planning 
done for the proposed Biomed system. Backup is available for 
air conditioning and power. There will be attractive features 
(e.g., full color printing) available in the Biomed system, 
and games will also be on the system. 

If the Biomed system is down for 24 hours, there 
will be no problem. If the Biomed system is down for 7 days 
or more, there will be a loss of productivity of $40,000 per 
day. Two weeks is the maximum acceptable downtime for this 
system. After that, a loss of confidence will occur and could 
cause possible loss of future funding. 

Since the data is used for biomedical research, 
compromises may be (but are not necessarily) related to a 
possible loss of human life through extended research time or 
improper authorization for human experimentation. Compromise 
could include: damage through error; unauthorized disclosure 
or modification; and unavailability of the Biomed system. 
There would be no monetary impacts (such as law suits), but 


compromise could result in failure to accomplish the agency 
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mission, improper interpretation of results, or loss of public 
confidence and future funding. 

The Office of Scientific Integrity has a strong 
policy on maintaining the integrity of scientific projects. 
Management allows group data passwords only if they are known 
by authorized users. Management has in-place resource 
protection measures which include: marking of all equipment; 
maintaining an active inventory of all hardware and software; 
and making personnel responsible for protection of government 
property. Despite this, there have been three cases of 
deliberate misuse of resources by authorized staff in the last 
year. The staff is trained in emergency procedures which 
include: evacuation procedures; CPR training; first-aid kits 
on each floor; and health facilities at each site. The Biomed 
system procedures will be written after the system is procured 
and the applications are developed. 

g. Management Philosophy and Concerns 

During the design phase of the Biomed system, an 
automated data processing security branch was developed to 
address and direct all security issues associated with the 
project. Top management envisioned this branch as a key 
contributor to the development of the new system. To fulfill 
this requirement, the branch established a risk management 
team of technical, administrative, management, and programming 


experts. The team's initial mission was to select a risk 
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management package to assist in system development. The CERTS 
DSS was used to select this package, and pairwise comparisons 
were made for all the criteria. For example, completeness was 
deemed to be more important, in varying degrees, than any 
other criteria. The rankings and DSS assigned weights for 
pairwise compared criteria are summarized in Table 22. 
Completeness was followed by credibility, consistency, 
validity, user interface, adaptability, and feasibility, 
respectively. 

The team’s primary concern in the choice of a risk 
management package is to ensure completeness. The package 
must take into consideration all relevant relationships and 
system elements of risk management. Since the Biomed system 
1S a new, under development system, top management is also 
concerned that the analysis considers all aspects of the 
system. Desired elements of coverage could include assets, 
threat agents, threat events, safeguards, vulnerabilities, and 
outcomes. This array of information is regarded as critical 
in the development of the DSS methodology and the satisfaction 
of the needs of the organization. The management desires that 
the relationships between the elements of risk are addressed 
in areas such as local and remote users, known activist 
threats, integrity of scientific projects, emergency 
Situations, backup situations, and contingency planning. 
Table 23 displays the rankings and the DSS assigned weights 


for the completeness subcriteria. 
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Management tends to view the credibility of a 
particular method or package with utmost importance when 
involved with sensitive data. The process used has a 
Significant bearing on the acceptability of its conclusions. 
The data produced in Biomedical research requires’ strong 
integrity protection to ensure that the results are correct. 
With the possibility of system compromises that could lead to 
the loss of human life, it is imperative that the risk package 
encompass all threats and vulnerabilities. The reliability of 
the method provides credence to those interpreting the output. 
If different results are returned using the same data on 
different occasions, the method will hold little plausibility 
for its users. Table 24 shows the rankings and the DSS 


assigned weights for the credibility subcriteria. 
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TABLE 23. COMPLETENESS 


Ranking 





With the strong requirement for maximizing 
productivity and ensuring that published results are correct, 
consistency is the next criteria emphasized by management. 
Scientific research provides an atmosphere of constant change 
and various risks. When an analyst is evaluating these risks, 
he has a tendency to make inferences based on what he 
remembers hearing or observing. A key component. of 
consistency, reliability, furnishes support for the reduction 
of subjectivity in the risk management process. Another 
concern in this process is controlling differences in 
interpretation. Interpretation is defined as the information 
being asked for versus what the product represents. A uniform 
set of terminology is a must between the analyst and the 
process. Table 25 depicts the rankings and the DSS assigned 
weights for the consistency subcriteria. 

The validity of a package is exposed to the 
numerous impacts that the risks impose on the data. Equal 
concern was expressed for the subcriteria of validity. To 
maintain relevancy of the results, it was felt that the 


results of the package must therefore relate to significant 
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TABLE 24. CREDIBILITY 


DSS_Assigned Weight 


| a intuitiveness | 524 
Reliability 












TABLE 25. CONSISTENCY 


DSS_Assigned Weight 


Reliability aie i 40 
Consistent Terminology | 450 


areas of need and also incorporate mandated security 
requirements. The user of the package must be able to control 
the level of detail being analyzed and must also be able to 
consider all aspects of the system. Table 26 presents the 
rankings and the DSS assigned weights for the validity 


subcriteria. 


TABLE 26. VALIDITY 


Ranking | subcriteria ____|_85 Assigned Weight _| 
SSS ee 


User interface was a minor concern, based on the 






experience and level of training of each member of the risk 


management team. The subcriteria of ease of use and 
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comprehension of underlying premises with methodology are a 
plus for this criteria. The team requires 24-hour phone 
support or a 1-800 number service. Table 27 provides the 
rankings and the DSS assigned weights for the user interface 
subcriteria. As the Biomed system procedures will be written 
after the system is procured, adaptability and feasibility are 
of less concern at the present time than other areas. Tables 
28 and 29 summarize the rankings and the DSS assigned weights 


for the adaptability and feasibility subcriteria. 


TABLE 27. USER INTERFACE 


ST hee a | = <n 
Mees 
Tea. 
We i 


Ls simplicity 062 


The CERTS DSS selected BDSS as the best risk 














management package for the Biomed scenario. The detailed 


results are shown in Appendix E, Templates 1 through 3. 


TABLE 28. ADAPTABILITY 


L088 Assigned Weight _ 


ee ne a ee eee 


Modi fiability 
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TABLE 29. FEASIBILITY 


DSS Assigned Weight 
practicality | 60 


2 2319 


As described in the first case, a sensitivity 







analysis can be performed if your priorities for the criteria 
change. If management decided they wanted a more portable 
package, additional weight would be applied to the 
adaptability criteria. With the newly assigned weight, the 
CERTS DSS would select RISKWATCH as the best risk management 
tool for the Biomed case. The sensitivity analysis is 
illustrated in Appendix E, Template 4. 
3. Case Scenario Three: Data Center 
The ABC Corporate Data Center supports the North 
American Operation, a subsidiary of United Corporation. The 
North American Operation has 400 full-time employees and is 
the fifth largest banking organization in the Northeast United 
States. The data center is responsible for processing 
checking accounts, savings deposits, loans, and savings 
certificates. Additional responsibilities include maintaining 
off-the-shelf personnel and management computer applications. 
a. Physical Environment 
Two buildings were converted for company use, ADP 


and Administration. The buildings are next to one another, 
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but not physically connected. The buildings are located in a 
large Northeastern American city on an average city street. 
A sidewalk runs along one side of each building. There is an 
adequately lit employee parking lot. The possibility of an 
earthquake in that area is low. 

The ADP Building is a 20 year old warehouse and 
houses the mainframe computer and the tape library. 
Conversion improvements consist of: raised flooring to 
accommodate cables and wiring; suspended acoustical tile 
ceiling to absorb sound and hide the overhead plumbing; power 
distribution upgrade; surge suppression; lighting; and air 
conditioning and heating. There are no under floor water 
detectors or temperature-humidity recording systems. The roof 
is in good shape, despite its age. Recently, water stains 
have been noticed in other parts of the building. The 
concrete floor below the raised floor was last cleaned when 
installed five years ago. 

The Administration Building was originally 
constructed as an ADP Center. Therefore, it is equipped with 
adequate environmental systems (similar to those of the 
current ADP building). When it was converted to its present 
use, an overhead sprinkler system was added to conform to fire 
codes. Neither building has an emergency backup generator. 


The power is supplied solely by the local power company. 
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b. Equipment 

The Data Center contains an XYZ-3100 Mainframe with 
Six tape drives, 12 disk drives, and 3 on-line printers, all 
located in the computer room of the ADP building. In 
addition, 20 terminals are located in an uncontrolled area of 
that building. These terminals are connected to the XYZ-3100 
Mainframe via the data communication system. Despite a 
constant workload, the system only operates at 60% capacity. 
This low capacity is attributed to poor employee performance, 
software failure, and unreliable equipment. The equipment 
repairs are performed on a ‘per incident’ basis by a 
contractor hired on that basis. No regular maintenance is 


performed on the hardware. The Administration Building has 


PCs, however, none of the PCs lock or are secured to the 


furniture. 
The dollar value of equipment is as follows: 
l mainframe computer @ $350,000 S$ 250,000 
6 tape drives 10,000 
12 disk drives 150,000 
30 personal computers 200,000 
1 communications controller 10,000 
2 modems 5,000 
4 multiplexers 4,000 
Other (paper, disks, printers, etc.) 100,000 
TOTAL value of equipment $ 829,000 


c. Personnel 
Background checks are not performed on new hires. 
Only the 20 data entry clerks and 10 computer operators (of 50 


employees) are considered essential to production operations. 
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However, excessive use of sick leave and a high rate of 
turnover is aeeproblem with these essential employees. 
Personnel shortage, the continuing need for overtime, and 
excess sick leave adds to the backlog of work, which must be 
made up at time-and-a-half rates. The loss of an operator or 
clerk results in recruitment fees and training costs for 
replacement personnel. The average level of experience for 
the system staff is two years. The average percentage of 
turnover in staff per year is 40%. 

No formal form of computer security training exists 
for personnel. The only existing training is for new data 
entry clerks on the performance of their jobs. There are no 
passwords for any system entry. 

d. Data Environment 

The Pay/Personnel and Financial /Management 
application systems are off-the-shelf and maintained by 
upgrades from the vendor. Company personnel trained on the 
software can make quick patches when necessary. These systems 
and data files constitute the critical work-load (80% of the 
total) of the Center. The rest of the work is general 
administration of the company, using standard business 
software. Backups are made once a week. These backups are 
stored in the tape library, with the original copy of the 
software. Backups are kept for three weeks before being 


recycled. 
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Data sensitivity 1S primarily based on its 
integrity requirement and is considered to be highly 
sensitive. The potential for loss, due to fraud or error, is 
high. The system controls 500 million dollars’ of 
disbursements annually, as well as ae payroll. The 
availability of the system is required for operations and 
employee productivity and has medium sensitivity, since it can 
be accessed manually. A confidentiality requirement exists 
for personnel data on the system. This data is classified as 
medium sensitive due to the Privacy Act stipulations placed on 
government contractors. 

e. Operating Systems 

The proprietary system software is supplied by the 
hardware vendor and provides no controls to limit access to 
software or data files. Copies of the system software may be 
obtained from the vendor at no charge and made operational in 
approximately eight hours. A standard operating procedure is 
to obtain a clean copy of the operating system from the vendor 
whenever the on-site OSS has become unusable. Whenever there 
is a production stoppage, the problem is located, fixed, and 
restarted at an appropriate point. Production problems are 
attributed to bad code or patching. The OSS has audit 
capabilities and that facility is occasionally used. 

In the event of an extended system unavailability, 


all data entry clerks and computer operators, as well as the 
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other ADP personnel, manually perform the computer's critical 
processing. All personnel are required to work an extended 
shift (10 hours) if an extended system unavailability occurs. 
Each hour of manual critical processing costs the company 
$7,500. 

The data communications system consists of one 
communications controller and one modem located in the ADP 
Building. These are connected by a single, underground line 
to one modem and four multiplexers (one primary and three 
secondary ) located in an uncontrolled area of the 
Administration Building. The communications equipment is five 
years old. As with the ADP equipment, the repairs to the data 
communications system are accomplished on a ‘per incident’ 
basis by a contractor. There is no regular maintenance on the 
communications system. 

f. Administration 

Security for the data is considered to be a low 
priority item, primarily due to budget. Documentation of 
operating and administrative procedures are located throughout 
the Center, but not kept up-to-date. The Center works one 
shift (eight hours) per day and normally generates $5,000 per 
hour in revenue. 

The system is "hacker friendly’ (e.g., no 
passwords, no warning screen). It is not difficult to ‘crash’ 


the network and enter the operating system, or other 
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applications. The company relies on software packages from 
the vendor to keep the system operational, and any enhancement 
the vendor chooses to put on the system is accepted. Any 
staff working outside normal hours have access to programming 
and editing facilities. The staff works on the system 
unsupervised. The organization does not rely on the 
communications equipment, therefore, its failure is not likely 
to result in complete stoppage. The network will continue to 
function in a degraded mode. 

The On-line Pay/Personnel system and 
Financial/Management information system are processed in the 
batch mode. All data entry is performed by data entry clerks. 
Updates to the master Pay/Personnel files are usually 
backlogged two to three days. All other data entry is often 
backlogged two weeks. Because of backlog, ten (of 20) data 
entry clerks and five (of ten) computer operators each work 
two hours per day (ten hours/week) overtime. As a result, 
computer operations are now scheduled for ten hours (eight 
hours plus two hours overtime) daily. All employees receive 
time-and-a-half for overtime work. Operating expenses 


(utilities, etc.) incurred from overtime amounts to $3,000 per 


hour. 
g. Management Philosophy and Concerns 
The converted data center was established to meet 
the immediate and expanding needs of the corporation. In 
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addition to meeting these needs, a small information 
technology (IT) group was developed. The group's primary 
responsibilities include; monitoring technological growth, 
Specialization of contemporary technology, and assisting the 
users with the significant shifts in the types of applications 
being automated. During the conversion, top corporate 
management tasked the IT group with all facets of ADP 
security. A risk assessment of corporate information systems 
is required annually. The CERTS DSS was selected to assist in 
this process. Pairwise comparisons were made of all the 
criteria in the DSS. For example, credibility was deemed to 
be more important than any other criteria. The rankings and 
DSS assigned weights on the evaluated criteria are presented 
in Table 30. 

The IT group is very concerned with output 
reliability and the merit of desired/required changes. This 
concern falls within the scope of credibility. When dealing 
with a data center, one needs to possess a strong sense of 
flexibility. Data in this environment is volatile and is 
constantly being altered. In this situation, data sensitivity 
1S primarily based on its integrity requirement and is 
considered highly sensitive. The management staff is seeking 
a risk management package or tool that will instill and 
maintain the confidence of the analyst throughout the entire 
process. The output of the process must have an obvious 


relationship to the data provided. 
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TABLE 30. DATA CENTER 


Gc. a ee ela 
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7 adaptability a 


The natural feel for the input, process, and output 











of a method is supported by the amount of information 
available to the user. This data center has numerous problems 
with unreliable equipment, software failures, and poor 
employee performance. These problems may result in a 
multitude of different risk conditions. The reliability of 
the package is critical to allow results to be repeated, and 
therefore, has a direct bearing on the credibility of a 
process. Table 31 shows the rankings and the DSS assigned 
weights for the credibility subcriteria. 

The validity of a risk management package closely 
follows the credibility criteria. As the processing method 
may be done manually oor with the current computer 
configuration, the package must be able to address the scope 
of the processing status. The tool must be able to provide 


the scope and detail required by the analyst to be valid. 
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TABLE 31. CREDIBILITY 





Because of the organization's tremendous dependency on the 
hardware vendor to solve problems, the relevancy of the 
results are critical. The desired results should provide 
categories of solutions rather than specific recommendations. 
Table 32 presents the rankings and the DSS assigned weights 
for the validity subcriteria. 

User interface is the next concern of the 
management. The average percentage of turnover in staff per 
year is 40%. This turnover rate is also reflected in the risk 
management staff. Therefore, management is searching for a 
package that does not require the user to grasp all the 
aspects of the process, but would allow an appreciation of the 
requirements of the system. Understanding the process 
contributes to the ease of use attribute. Again, due to such 
a high turnover rate, a consistent interface must exist that 
allows the analyst to concentrate on his task rather than on 
the process itself. The group is seeking a package with well 
written documentation, on-site training, on-site repair, and 
24 hour phone support. Table 33 provides the rankings and the 


DSS assigned weights for the user interface subcriteria. 
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The criterion of feasibility is of less concern to 
management because the availability of the data is accessible 
both within and external to the organization. The cost of 
gathering the required data has been determined to be minimal. 
A conscientious decision to invest the necessary effort and 
time to accomplish this task has been made. Table 34 depicts 
the rankings and the DSS assigned weights for the feasibility 
subcriteria. 

The remaining three criteria: consistency, 
completeness,and adaptability are considered less important 
than the first four criteria. The IT group at this time 
prefers to focus on the first four criteria as the major 


requirement for the system. 


TABLE 32. VALIDITY 


Ce 
- ee See eee eee 


The three remaining criteria were ranked in the 









order of consistency, completeness, and then adaptability. 
Due to the less’ significance of these criteria, all 
subcriteria within each criteria were determined to be of 


equal importance. Tables 35, 36, and 37 summarize the 
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rankings and the DSS assigned weights of the each criteria, 
respectively. 

The CERTS DSS selected LAVA as the best risk 
management package for the data center scenario. The detailed 


results are shown in Appendix F, Templates 1 through 3. 


TABLE 33. USER INTERFACE 


08S Assigned Weight _ 


Understandable 
| 2 ease of use | tt 


[——«[simenicity fase 
| sd error Handling | 





TABLE 34. FEASIBILITY 


DSs_Assigned Weight _ 
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TABLE 35. CONSISTENCY 


_Dss a Weight _ 
meena Reliability 


Consistent 
| Terminology _ 
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TABLE 36. COMPLETENESS 
DSS Assigned Weight 
Attributes 













TABLE 37. ADAPTABILITY 
DSS_Assigned Weight 
Modifiability | 500, 
ee | 





As with the previous two case studies, a 
sensitivity analysis can be performed for the data center. If 
the IT group determines that the completeness of a package 
needs more emphasis, then the CERTS DSS would select BDSS as 
the best risk management tool for the data center. The 


sensitivity analysis is illustrated in Appendix F, Template 4. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 
1. CERTS Decision Support System 

Garrabrants and Ellis developed an approach, CERTS, 
which would select the best risk management tool for a given 
organizational situation. While this approach is beneficial 
to organizations, it is, in its current form, very complex and 
time consuming to apply. It requires answering an extensive 
series of questionnaires for each risk management package that 
an organization iS considering. Additionally, extensive 
calculations are required to synthesize the results of the 
questionnaires into a suitability index that helps the 
Organization to select the best risk management package. 
Garrabrants and Ellis' approach also offers no way to weight 
certain metrics of the questionnaire which are more important 
to the organization selecting the risk management package. 

Combining CERTS with the AHP approach into an 
automated Decision Support System alleviates many of the above 
weaknesses. First, it is simple and easy to use. Second, the 
decision support system does not require the analysts of the 
organization to become experts in all the risk management 
packages under consideration. The analysis of the risk 


management packages with respect to the detailed 


Us 


subsubcriteria 1s already completed and incorporated in the 
decision support system. Third, all calculations are done 
automatically, thus saving a considerable amount of time and 
effort. 

The CERTS Decision Support System is based on T.L. 
Saaty's Analytical Hierarchy Process (AHP). Under this 
approach, the decision of selecting the best risk management 
package is modeled as a hierarchy. The top level is 
considered the goal, and the subsequent levels represent the 
criteria, subcriteria, and subsubcriteria with each succeeding 
level being a refinement of the higher level. Finally, the 
leaves of the hierarchy represent the alternatives, which are 
the risk management packages under consideration. The basis 
for making the selection is the pairwise comparison of the 
criteria, subcriteria, and subsubcriteria. In this way, 
organizations can place more importance on certain criteria, 
subcriteria, or subsubcriteria which they deem more important 
for their particular situation. After all pairwise 
comparisons are made, the decision support system selects the 
best risk management package for that given situation. 

2. Case Studies 

The case studies used for applying the CERTS Decision 
Support System were based on cases that the National Institute 
of Standards and Technology's Risk Management Laboratory used 


in testing risk management packages. All aspects of the cases 
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were based on hypothetical organizations while the management 
Philosophy and concerns were the authors’ inferences and 
conclusions based on the description of each particular case. 

These inferences and concluSions were then used to 
make the pairwise comparisons in the CERTS Decision Support 
System. Depending on the requirement of each case, the 
decision support system selected a risk management package to 
best meet the needs of each organization. 

When an organization, through pairwise comparison, 
establishes the importance of each criteria, subcriteria, or 
subsubcriteria, the CERTS Decision Support System assigns 
weights to each criteria and selects the best risk management 
package for the organization. 

As each risk management package has its strengths and 
weaknesses, and each organization has different requirements, 
there is no single package that could be designated as the 
package of choice for all organizations. Since the strengths 
and weaknesses of each package under consideration are 
incorporated in the DSS, pairwise comparisons based on the 
organizations's requirements, will result in selecting the 


best package for the organization. 


B. RECOMMENDATIONS 
The CERTS DSS needs to include more risk management 
packages, at the leaf nodes of the hierarchy, to make the tool 


beneficial for organizational usage. This study used only 
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three risk management packages in developing the DSS. There 
are numerous risk management packages available for 
organizations, and for the DSS to be effective, these packages 
need to be analyzed and placed in the hierarchy so that the 
package selected by the DSS is the best available package. 
The criteria, subcriteria, and subsubcriteria of the DSS 
need to be refined further. The metrics from Garrabrants and 
Ellis’ thesis were modified for this study, but further 
refinement is necessary to make the DSS a more effective tool. 
Validation of the CERTS DSS needs to be accomplished on 
actual case studies. This study was completed by using 
hypothetical cases. To determine the effectiveness of the 
DSS, real life case situations should be used for evaluation. 
Elimination oof infeasible alternatives should _ be 
accomplished before the DSS is used by an organization. For 
example, an organization wants to spend no more than $1,000 
for the risk management package. The system should screen out 
those risk management packages costing over $1,000 and 
establish the DSS only with alternatives meeting the 


requirements. 
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APPENDIX A. SUBSUBCRITERIA OF THE DECISION SUPPORT SYSTEM 


TEMPLATE 1. 


Consistency Criteria 


a ——— SS Se SS Saar SS ae a 


Subsubcriteria 


Reliability Reducing the introduction of personal 
bias 


Reducing the impact of uncertainty 


| Consistent Establishing standard language 


Terminology Defining method for the user 
Requesting input in designated units 


Requesting input unambiguously 




















= = ene © — wee ee rn ae | 
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TEMPLATE 2. 


<2 " User Interface Hierarchy 
Subsubcriterla 


Error Readily identifying data entry errors 


Handling Facilitating the handling of data entry 
errors 





Being insensitive to insignificant data 
accuracy errors 

Requiring smaller knowledge base to operate 
the process 


Simplicity 
Mitigating complex relationships for the 
user 
Defining problem domain 
Not requiring special training to interpret 
reports 
Being well structured and logically 
sequential 


Understand 
ability 


Explaining relationships between phases and 
iterations 


Identifying decision points clearly 


Not requiring special training to operate 

Ease of Use | Having standardized interface 
Differentiating one iteration clearly from 
others 





Support Developer providing support for product 
Providing technical support by phone 
Providing written documentation 





Providing on site training 
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TEMPLATE 3. 


Adaptability Hierarchy 


Subsuberiteria 
Portability Applying across system configurations 
Applying across processing methods | 





Applying across different environments 


Applying across all phases of system 
life cycle 


Modifiability Retaining inputs in original form 


Segmenting calculations by identifiable 






partitions 





Modifying software package 
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TEMPLATE 4. 


Feasibility Hierarchy 
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Availability 
Practicality 


Subsubcriteria 











Requiring expert opinion for methods 
internal to the organization 


Required data being internal to the 
organization 


Collection of data being convenient at 
the scope desired 


Allowing input in a variety of forms 


Performing the process by available staff 





Time being available to perform the 
process 


Obtaining precision economically 


User selecting amount of detail 
Bounding detail at the level desired 
Analyzing all data aspects of the system 


Analyzing procedural aspects of the 
system 


Analyzing personnel aspects of the system 


Analyzing communication aspects of the 
system 





! 
’ 
‘ 
1 
t 
4 
{ 
t 
t 
‘ 


Analyzing environment of the system 
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TEMPLATE 5. 


Completeness Hierarchy 
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Subsubcriteria 


Scope User selecting amount of detail 
Bounding detail at the level desired 
Analyzing all data aspects of the system 
Analyzing procedural aspects of the system 
Analyzing personnel aspects of the system 
Analyzing communication aspects of the 
system 
Analyzing environment of the system 

Elements Comprehensively considering assets 
Comprehensively considering threat agents 
Comprehensively considering threat events 
Comprehensively considering safeguards 
Comprehensively considering vulnerabilities 
Considering outcomes 


Considering asset values 






Elements 


Attributes Considering potency of threat agents 






Considering undesirability of threat events 
Considering effectiveness of safeguards 
Considering severity of outcomes 


Considering probabilities of the occurrence 
of threat events 
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TEMPLATE 6. 


Validity Hierarchy 
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Subsubcriteria 


Relevancy Expressing results in terms of solutions 
rather than specifics 
Results relating to significant areas of 
need 
Results fulfilling mandated requirements 
and regulations 
Output results being qualitative 
Output results being quantitative 
User selecting amount of detail 
Bounding detail at the level desired 
Analyzing all data aspects of the system 
Analyzing procedural aspects of the system 
Analyzing personnel aspects of the system 
Analyzing communication aspects of the 
system 
Analyzing environment of the system 


Practicality [| Allowing input in a variety of forms 
Performing the process by available staff 
Time being available to perform the 
process 
fee OPEAINing precision economically 
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TEMPLATE 7. 
Seared aE 


Subsubcriteria 





Intuitiveness Delineating the relationships between 
the results 


Output being a perceivable relationship 
with the inputs 


Analyzing all data aspects 
Analyzing procedural aspects 
Analyzing personnel aspects 
Analyzing communication aspects 
Analyzing environment aspects 


Reliability Reducing the introduction of personal 
bias 


Reducing the impact of uncertainty 
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APPENDIX B. CERTS DSS HIERARCHY 


TEMPLATE 1. 


Goal, Criteria, and Subcriteria 
A COMPARATIVE EVALUATION METHOD FOR RISK MANAGEMENT TOOLS 
! GOAL ! 
4 | 
i 1-000! 
’ 
' 1 
' !tuSEIFACE! ‘ADAPTITY! !FEASBITY! !COMPLETE! ! VALIDITY! '!CREDIBTY! 
i] 1 i] 1 i] ] | i] i] i] | i] 
eel 4s! Swi wormrac! ! leone!) LnOel4s! ! L 0.143! ! L 0.143! 
!=ERORHAND !-PORTABLE !-AVAILBTY !-SCOPE C !-RELEVNCY !-INTUITVE 
L5@.200 “beso ) ! I@esaseen L 6.333 ! t 0.333 ! L 0.500 
'-SIMPLE  !-MODIFITY !-PRACTICL !-ELEMENTS !-SCOPE VV !-RELIABTY 
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!-EASE USE '-SCOPE F !-ATTRIBUT !-PRACTLTY 
L 0.200 (sin aa3° 9 ie Oessag ee! -L 6.333 
!-UNDESTND 
L 0.200 
'-SUPPORT 
L 0.200 


ADAPTITY 
ATTRIBUT 
AVAILBTY 
COMPLE TE 
CONSISTY 
CREOIBTY 
CSTTTERM 
EASE USE 
ELEMENTS 
ERORHAND 
FEASBITY 
INTUITVE 
MODIFITY 
PORTABLE 
PRACTICL 
PRACTLTY 
RELEVNCY 
RELIABTY 
BELIBITY 
SCOPE C 
SCOPE F 

SCOPE V 
SIMPLE 

SUPPORT 

UNDE STND 
USE IFACE 
VALIDITY 
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STRUCTURE OF METHOD CAN BE APPLIED TO VARIOUS SYSTEMS 
DETERMINATION OF OUTCOMES OR CONSEQUENCES THAT COULD RESULT 
DISTINGUISHES BETWEEN INTERNAL AND EXTERNAL DATA 

PROVIDING COMPLETE COVERAGE OF ALL RISK MANAGEMENT PROBLEMS 
ABILITY TO DUPLICATE THE RESULTS OF THE PROCESS 

CONCLUSIONS ARE ACCEPTABLE 

UNIFORM SET OF TERMINOLOGY WITHIN THE SYSTEM 

A PROCESS THAT IS WELL STRUCTURED AND LOGICALLY SEQUENTIAL 
THREE CENTRAL ELEMENTS OPERATE TO DETERMINE THE RISK OF SYSTEM 
IDENTIFYING INPUT ERRORS AND RESOLUTION OF THEM 

AMOUNT OF EFFORT AND COST TO OBTAIN THE NECESSARY DATA 
RESULTS SHOULD INSTILL AND MAINTAIN CONFIDENCE OF ANALYST 
ASSISTS ANALYSTS IN EXAMINING ALTERNATIVES OR OPTIONS 
ABILITY TO APPLY THE PROCESS ACROSS A VARIETY OF SYSTEMS 
CONCERNED WITH THE ECONOMICS OF GATHERING THE REQUIRED DATA 
FEASIBILITY OF ACCOMPLISHING DESIRED TASK 

RESULTS ARE MEANINGFUL TO THE SYSTEM 

ABILITY TO OBTAIN REPEATABLE RESULTS 


“OBJECTIVITY OR THE REDUCTION OF SUBJECTIVITY IN THE PROCESS 


THE LEVEL OF DETAIL OF ANALYSIS / CONSIDER ALL ASPECTS OF SYSTEMS 
INFLUENCES THE ACCEPTABILITY AND USEFULNESS OF A METHOD 
DETERMINES THE EXTENT OF THE OETAIL USEO BY THE PROCESS 
COMPLEXITY OF THE PROCESS IS CONCEALED W/O OBSCURING THE PROCESS 
SUPPORT PROVIDED BY THE PROGRAM AND/OR THE DEVELOPER 

ABILITY TO COMPREHEND THE UNDERLYING PREMISE THAT SUPPORTS METHOD 
THE EFFORT NECESSARY BY OPERATOR TO UNDERSTAND COMPLETE SYSTEM 
RESULTS OF THE PROCESS REPRESENT REALITY 


LOCAL PRIORITY: PRIORITY RELATIVE TO PARENT 


83 


BIAS 
CONSISTY 
CSTTTERM 
DEF INED 
INPUTRQT 
LANGUAGE 
RELIBITY 
REQUEST 
UNCERTTY 
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TEMPLATE 2. 


Criteria, Subcriteria, and Subsubcriteria 
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'CONSISTY! 0 0 0 0 0 0 
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'-BIAS !-LANGUAGE 
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!-REQUEST 
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MECHANISM TO REDUCE THE INTRODUCTION OF PERSONAL BIAS 
ABILITY TO DUPLICATE THE RESULTS OF THE PROCESS 

UNIFORM SET OF TERMINOLOGY WITHIN THE SYSTEM 

METHOD’S ELEMENTS DEFINED FOR THE USER 

INPUT REQUESTED UNAMBIGUOUS 

STANDARD LANGUAGE ESTABLISHED 

OBJECTIVITY OR THE REDUCTION OF SUBJECTIVITY IN THE PROCESS 
METHOD REQUEST INPUT IN DESIGNATED UNITS 

PROVIDE A MECHANISM THAT REDUCES THE IMPACT OF UNCERTAINTY 


LOCAL PRIORITY: PRIORITY RELATIVE TO PARENT 
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COMPREHD 
DATAENTY 
DOCUMENT 
DOMAIN 
EASE USE 
ERORHAND 
IDENT 
INTERFAC 
ITERATIO 
KNOWBASE 
PHASES 
PHONE 
POINTS 
PREMISE 
PROCESS 
PRODUCT 
RELATION 
RELEVANT 
RPTTRAIN 
SENSITVE 
SIMPLE 
SITETRNG 
SUPPORT 
TERMS 
TRAINING 
UNDESTND 
USEIFACE 


ti 


TEMPLATE 2. (continued) 
O 
! 
' ! roo roof 
O ‘USEIFACE! 0 0 0 O 
i i 
' L 0.143! 
4 
‘ 

' ! ! ! ! 
'ERORHAND! !SIMPLE ! !'EASE USE! !UNDESTND! !SUPPORT ! 
i i] ( i] i] i] i ] ] i 
' L 0.200! ! L 60.200! ! L 0.200! ! L 0.200! ! L 0.200! 
'-IDENT '-KNOWBASE !-INTERFAC !-PREMISE !-PRODUCT 
' tL 0.333 !L 0.200 ! L 0.250 ! L 0.200 ! L 0.250 
'-DATAENTY !-RELATION !-ITERATIO !-COMPREHD !-PHONE 
'L 0.333 ! L 0.200 ! L 0.250 ! L 0.200 ! L 0.250 
'-SENSITVE !-DOMAIN !-PROCESS !-TERMS ! -DOCUMENT 
'L 0.333 ! L 0.200 ! L 0.250 ! L 0.200 ! L 0.250 

'=TRAINING !-RELEVANT !-PHASES !-SITETRNG 
'L 0.200 ! L 0.250 ! L 0.200 ! L 0.250 
!—-RPTTRAIN '-POINTS 
' L 0.200 ! L 0.200 


COMPREHENDIBLE PREMISE 

THE HANDLING OF DATA ENTRY ERRORS 

DEVELOPER PROVIDES WRITTEN DOCUMENTATION OF PROGRAM 

PROBLEM DOMAIN WELL DEFINED 

A PROCESS THAT IS WELL STRUCTURED AND LOGICALLY SEQUENTIAL 
IDENTIFYING INPUT ERRORS AND RESOLUTION OF THEM 

DATA ENTRY ERROR IDENTIFICATION 

STANDARDIZED INTERFACE 

ITERATION CLEARLY DIFFERENTIATED FROM ANOTHER 

SMALLER KNOWLEDGE BASE REQUIRED TO OPERATE THE PROCESS 

R’SHIPS BETWEEN ELEMENTS EXPLAINED BETWEEN PHASES OR ITERATIONS 
TECHNICAL SUPPORT PROVIDED BY PHONE CONVERSATION 

DECISION POINTS CLEARLY IDENTIFIED 

UNDERLYING PREMISE EXPLAINED 

PROCESS WELL STRUCTURED AND LOGICALLY SEQUENTIAL 

DEVELOPER PROVIDES SUPPORT FOR HIS PRODUCT /PROGRAM 

COMPLEX RELATIONSHIPS MITIGATED FOR THE USER 

INFORMATION REQUESTED OF THE USER RELEVANT 

TRAINING TO INTERPRET REPORTS 

INSENSITIVE TO INSIGNIFICANT DATA ACCURACY ERRORS 

COMPLEXITY OF THE PROCESS IS CONCEALED W/O OBSCURING THE PROCESS 
DEVELOPER PROVIDES TRAINING ON SITE 

SUPPORT PROVIDED BY THE PROGRAM AND/OR THE DEVELOPER 
TERMS UNAMBIGUOUSLY DEFINED 

SPECIAL TRAINING REQUIRED TO OPERATE/UNDERSTAND PROGRAM 


ABILITY TO COMPREHEND THE UNDERLYING PREMISE THAT SUPPORTS METHOD 


THE EFFORT NECESSARY BY OPERATOR TO UNDERSTAND COMPLETE SYSTEM 


LOCAL PRIORITY: PRIORITY RELATIVE TO PARENT 
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ADAPTITY 
ENVIRONS 
EIRECTEC 
MODIFITY 
ORIGINAL 
PARTITON 
PORTABLE 
PRO METH 
SYS CON 

TAILOR 


E 


TEMPLATE 2. (continued) 


O 
' 
ae ! poroboy 
0 O !ADAPTITY! 0 00 0 
8 Q 
! tL 0.143! 
q 
! ! 
!PORTABLE! !MODIFITY! 
) J uy ) 
! tL 0.500! ! L 0.500! 
'-SYS CON !-ORIGINAL 
!t 0.250 ! L 0.333 
'-PRO METH !-PARTITON 
! tL 0.250 ! L 0.333 
!-ENVIRONS !-TAILOR 
!t 0.250 ! L 0.333 
'~LIFECYLC 
! L 0.250 


STRUCTURE OF METHOD CAN BE APPLIED TO VARIOUS SYSTEMS 

PROCESS APPLIED ACROSS ENVIRONMENTS ( TERMINAL/DISTRIBUTED ) 
PROCESS APPLIED ACROSS ‘ALL PHASES OF THE SYSTEM LIFE CYCLE 
ASSISTS ANALYSTS IN EXAMINING ALTERNATIVES OR OPTIONS 

INPUT VALUES RETAINED IN THEIR ORIGINAL FORM 

CALCULATIONS SEGMENTED BY IDENTIFIABLE PARTITIONS 

ABILITY TO APPLY THE PROCESS ACROSS A VARIETY OF SYSTEMS 

PROCESS APPLIED ACROSS PROCESSING METHODS (BATCH/INTERACTIVE ) 
PROCESS IS APPLIED ACROSS SYSTEM CONF IGURATIONS9S( MAIN/MINI/MICRO ) 
SOFTWARE PACKAGE CAN BE MODIFIED 


LOCAL PRIORITY: PRIORITY RELATIVE TO PARENT 
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ACCOMPLH 
AVAILBTY 
BOUND 
CNVENINT 
COMM 
DASPECTS 
DETAIL 
ENVIROMT 
FEASBITY 
INTERNAL 
OPINION 
PERFORM 
PERSONEL 
PRACTICL 
PRECISON 
PROCEDUR 
SCOPE F 
VARIETY 


L 


TEMPLATE 2. (continued) 


O 
‘ 
too ! por 
© 0 O !FEASBITY! 0 @ 
4 A 
! L 0.143! 
‘ 
‘ 

! 
'AVAILBTY! !PRACTICL! !SCOPE F ! 
i i i i) i q 
! L 0.333! ! L 0.333! ! L 0.333! 
'-OPINION !-VARIETY !-DETAIL 
!L 0.333 ! L 0.250 ! L 0.143 
'-INTERNAL !-PERFORM !-BOUND 
!L 0.333 ! L 0.250 ! L 0.143 
!-CNVENINT !-ACCOMPLH !~-DASPECTS 
!L 0.333 !L 0.250 ! L 0.143 

!~PRECISON !-PROCEDUR 

WiG-250 !' L 0.143 
! -PERSONEL 
!L 0.143 
!—-COMM 
!L 0.143 
!-ENVIROMT 
!L 0.143 


TIME IS AVAILABLE TO PERFORM THE PROCESS 

DISTINGUISHES BETWEEN INTERNAL AND EXTERNAL DATA 
METHOD BOUNDS THE DETAIL AT THE LEVEL DESIRED 

DATA COLLECTION CONVENIENT AT THE SCOPE DESIRED 

ALL COMMUNICATIONS ASPECTS OF THE SYSTEMS ARE ANALYZED 
ALL DATA ASPECTS OF THE SYSTEM ARE ANALYZED 

AMOUNT OF DETAIL USER SELECTABLE 

THE ENVIRONMENT THAT THE SYSTEM RESIDES IN IS ANALYZED 
AMOUNT OF EFFORT AND COST TO OBTAIN THE NECESSARY DATA 
ALL DATA REQUIRED IS INTERNAL TO THE ORGANIZATION 
EXPERT OPINION REQUIRED FOR THE METHODS INTERNAL 
AVAILABLE STAFF PERFORMS THE PROCESS 

ALL PERSONNEL ASPECTS OF THE SYSTEM ARE ANALYZED 
CONCERNED WITH THE ECONOMICS OF GATHERING THE REQUIRED DATA 
PRECISION CAN BE OBTAINED ECONOMICALLY 

THE PROCEDURAL ASPECTS OF THE SYSTEM ARE ANALYZED 
INFLUENCES THE ACCEPTABILITY AND USEFULNESS OF A METHOD 
ALLOWS INPUT DATA IN A VARIETY OF FORMS 


TO ORGANIZATION 


LOCAL PRIORITY: PRIORITY RELATIVE TO PARENT 
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ASSETS 
ATTRIBUT 
BOUND 
COMM 
COMPLETE 
DASPECTS 
DETAIL 
EFFCTVNS 
ELEMENTS 
ENVIROMT 
OUTCOMES 
PERSONEL 
POTENCY 
PROBABTY 
PROCEDUR 
SAF EGURD 
SCOPE C 
SEVERITY 
T AGENTS 
T EVENTS 
UNDESIRE 
VALUES 
VULNERBY 


L 


TEMPLATE 2. (continued) 


O 
' 
a ' ss 
0 0 O O- !COMPLETE! 0 O 
t i 
! L 0.143! 
‘ 
! 

! ! ! 
SCOPE C ! !ELEMENTS! !ATTRIBUT! 
i i i i J § 
! L 0.333! ! L 0.333! ! L 0.333! 
'-DETAIL !-ASSETS  !-VALUES 
'L 0.143 ! L 0.167 ! L 0.167 
!-BOUND '-T AGENTS !-POTENCY 
!L 0.143 ! Lb 0.167 !L 0.167 
!-DASPECTS !-T EVENTS !-UNDESIRE 
'L 0.143 ! L 0.167 ! L 0.167 
!-PROCEDUR !-SAFEGURD !-EFFCTVNS 
iL 0.143 “ieoeie7 190.167 
!-PERSONEL !-VULNERBY !-SEVERITY 
!L 0.143 ! L 0.167 ! L 0.167 
!-COMM '~QUTCOMES !-PROBABTY 
'L 0.143 ! L 0.167 ! L 0.167 
!-ENVIROMT 
Leon as 


COMPREHENSIVELY CONSIDER ASSETS 

DETERMINATION OF OUTCOMES OR CONSEQUENCES THAT COULD RESULT 
METHOD BOUNDS THE DETAIL AT THE LEVEL DESIRED 

ALL COMMUNICATIONS ASPECTS OF THE SYSTEMS ARE ANALYZED 
PROVIDING COMPLETE COVERAGE OF ALL RISK MANAGEMENT PROBLEMS 
ALL DATA ASPECTS OF THE SYSTEM ARE ANALYZED 

AMOUNT OF DETAIL USER SELECTABLE 

SAFEGUARD EFFECTIVENESS IS CONSIDERED 

THREE CENTRAL ELEMENTS OPERATE TO DETERMINE THE RISK OF SYSTEM 
THE ENVIRONMENT THAT THE SYSTEM RESIDES IN IS ANALYZED 
CONSIDER OUTCOMES 

ALL PERSONNEL ASPECTS OF THE SYSTEM ARE ANALYZED 

POTENCY OF A THREAT AGENT IS CONSIDERED 

PROBABILITY OF THE OCCURENCE OF A THREAT EVENT IS CONSIDERED 
THE PROCEDURAL ASPECTS OF THE SYSTEM ARE ANALYZED 
COMPREHENSIVELY CONSIDER SAFEGUARDS 

THE LEVEL OF DETAIL OF ANALYSIS / CONSIDER ALL ASPECTS OF SYSTEMS 
SEVERITY OF OUTCOME IS CONSIDERED 

COMPREHENSIVELY CONSIDER THREAT AGENTS 

COMPREHENSIVELY CONSIDER THREAT EVENTS 

UNDESIRABILITY OF A THREAT EVENT IS CONSIDERED 

ASSET VALUES CONSIDERED 

COMPREHENSIVELY CONSIDER VULNERABILITIES 


LOCAL PRIORITY: PRIORITY RELATIVE TO PARENT 
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TEMPLATE 2. (continued) 


O 
‘ 
eo eee ! 
0 0 0 0 O- !VALIDITY! O 
0 0 
!L 0.143! 
' 
! ! ! 
'RELEVNCY! !SCOPE V ! !PRACTLTY! 
U t t § 0 ) 
! L 0.333! ! L 0.333! ! L 0.333! 
'-SOLUTION !-DETAIL !-VARIETY 
!L 0.200 ! L 0.143 ! L 0.250 
'-SIGNIFCT !-BOUND !-PERFORM 
!L 0.200 ! 4 0.143 ! L 0.250 
!-REQURMTS !-DASPECTS !-ACCOMPLH 
!L 0.200 ! L 0.143 ! L 0.250 
'-QUALITY !-PROCEDUR !-PRECISON 
!L 0.200 ! L 0.143 ! L 0.250 
'-QUANTITY !-PERSONEL 
!L 0.200 ! L 0.143 
1-COMM 
'L 0.143 
!-ENVIROMT 
' L 0.143 
ACCOMPLH --- TIME IS AVAILABLE TO PERFORM THE PROCESS 
BOUND --- METHOD BOUNDS THE DETAIL AT THE LEVEL DESIRED 
COMM --- ALL COMMUNICATIONS ASPECTS OF THE SYSTEMS ARE ANALYZED 
DASPECTS --- ALL DATA ASPECTS OF THE SYSTEM ARE ANALYZED 
DETAIL --- AMOUNT OF DETAIL USER SELECTABLE 
ENVIROMT --- THE ENVIRONMENT THAT THE SYSTEM RESIDES IN IS ANALYZED 
PERFORM --- AVAILABLE STAFF PERFORMS THE PROCESS 
PERSONEL --- ALL PERSONNEL ASPECTS OF THE SYSTEM ARE ANALYZED 
PRACTLTY --- FEASIBILITY OF ACCOMPLISHING DESIRED TASK 
PRECISON --- PRECISION CAN BE OBTAINED ECONOMICALLY 
PROCEDUR --- THE PROCEDURAL ASPECTS OF THE SYSTEM ARE ANALYZED 
QUALITY --- DESIRED OUTPUT RESULTS ARE QUALITATIVE 
QUANTITY --- DESIRED OUTPUT RESULTS ARE QUANTIATIVE 
RELEVNCY --- RESULTS ARE MEANINGFUL TO THE SYSTEM 
REQURMTS --- FULFILLS MANDATED REQUIREMENTS OR REGULATIONS 
SCOPE V --- DETERMINES THE EXTENT OF THE DETAIL USED BY THE PROCESS 
SIGNIFCT --- RESULTS RELATE TO SIGNIFICANT AREAS OF NEED 
SOLUTION --- RESULTS ARE EXPRESSED IN TERMS OF SOLUTIONS RATHER THAN SPECIFICS 
VALIDITY --- RESULTS OF THE PROCESS REPRESENT REALITY 
VARIETY --- ALLOWS INPUT DATA IN A VARIETY OF FORMS 
L --- LOCAL PRIORITY: PRIORITY RELATIVE TO PARENT 
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BIAS 

COMM 

CREDIBTY 
DASPECTS 
DELINETE 
ENVIROMT 
INTUITVE 
PERCEIVE 
PERSONEL 
PROCEDUR 
RELIABTY 
UNCERTTY 


L 


TEMPLATE 2. (continued) 


0 
: 
i 


0 0 0 0 0 0 !CREDIBTY! 


L 0.143! 


1 
! ! 
!INTUITVE! !RELIABTY! 
i] i f b 
! L 0.500! ! L 0.500! 


!-DELINETE !-BIAS 
£bLO.143 ! L O.500 
!-PERCEIVE !-UNCERTTY 
tL 0.143 $$! L.OFSo0 
!-DASPECTS 

!L 0.143 

!-PROCEDUR 

! L 0.143 

!-PERSONEL 

! L 0.143 
!-COMM 
!L 0.143 
!-ENVIROMT 
!L 0.143 


MECHANISM TO REDUCE THE INTRODUCTION OF PERSONAL BIAS 

ALL COMMUNICATIONS ASPECTS OF THE SYSTEMS ARE ANALYZED 
CONCLUSIONS ARE ACCEPTABLE 

ALL DATA ASPECTS OF THE SYSTEM ARE ANALYZED 

DELINEATES THE RELATIONSHIPS BETWEEN THE ELEMENTS 

THE ENVIRONMENT THAT THE SYSTEM RESIDES IN IS ANALYZED 
RESULTS SHOULD INSTILL AND MAINTAIN CONFIDENCE OF ANALYST 
OUTPUT HAS A PERCEIVALBLE RELATIONSHIP WITH THE INPUTS 
ALL PERSONNEL ASPECTS OF THE SYSTEM ARE ANALYZED 

THE PROCEDURAL ASPECTS OF THE SYSTEM ARE ANALYZED 

ABILITY TO OBTAIN REPEATABLE RESULTS 

PROVIDE A MECHANISM THAT REDUCES THE IMPACT OF UNCERTAINTY 


LOCAL PRIORITY: PRIORITY RELATIVE TO PARENT 
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APPENDIX C. DECISION SUPPORT SYSTEM ASSIGNED WEIGHTS FOR 
ALTERNATIVES 


TEMPLATE 1. 


Criteria: Consistency 
Subcriteria: Reliability 
Subsubcriteria: Reducing the Introduction of Personal Bias 






BDSS .163 


LAVA . 540 


Subsubcriteria: Reducing the Impact of Uncertainty 


BDSS - 540 
LAVA 2 9 


RISKWATCH .163 
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TEMPLATE 2. 


Criteria: Consistency 
Subcriteria: Consistent Terminology 
Subsubcriteria: Establishing Standard Language 


BDSS Be i I) 


LAVA 7oo0 


RISKWATCH .333 WY 





Subsubcriteria: Defining Method for the User 


BDSS .143 
LAVA ~714 


RISKWATCH .143 





Subsubcriteria: Requesting Input in Designated Units 


BDSS - 444 
LAVA - 444 


RISKWATCH .111 
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TEMPLATE 2. (continued) 


Subsubcriteria: Requesting Input Unambiguously 


BDSS .558 
LAVA oi 22 


RISKWATCH .320 





93 


TEMPLATE 3. 


Criteria: User Interface 
Subcriteria: Error Handling 
Subsubcriteria: Readily Identifying Data Entry Errors 


BDSS - 383 


LAVA So's bia “i 


RISKWATCH .333 YW 





Subsubcriteria: Facilitating the Handling of Data Entry 
Errors 

BDSS .297 

LAVA .540 


RISKWATCH .163 





Subsubcriteria: Being Insensitive to Insignificant Data 
Accuracy Errors 


BDSS 333 


LAVA Jo 


RISKWATCH .333 ZZ 
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TEMPLATE 4. 


Criteria: User Interface 
Subcriteria: Simplicity 
Subsubcriteria: Requiring Smaller Knowledge Base to Operate 
the Process 


BDSS -558 


LAVA 222 


RISKWATCH .320 WY 





Subsubcriteria: Mitigating Complex Relationships for the User 


BDSS . 344 


LAVA 23S 


RISKWATCH .344 WY 





Subsubcriteria: Defining Problem Domain 


BDSS 355 


LAVA 3353 eee 


RISKWATCH .333 KY) 





25 


TEMPLATE 4. (continued) 


Subsubcriteria: Not Requiring Special Training to Operate 


BDSS ~35s 
LAVA . 335 


RISKWATCH .333 





Subsubcriteria: Not Requiring Special Training to Interpret 
Reports 


BDSS 2359 


LAVA 335 


RISKWATCH .333 WY 
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TEMPLATE 5. 


Criteria: User Interface 

Subcriteria: Ease of Use 
Subsubcriteria: Having Standardized Interface 
BDSS .163 
LAVA .540 


RISKWATCH .297 





Subsubcriteria: Differentiating One Iteration Clearly From 
Others 


BDSS . 163 





LAVA . 540 : 


RISKWATCH .297 ‘ 


Subsubcriteria: Being Well Structured and Logically 
Sequential 


BDSS 29'S 


LAVA es) 


RISKWATCH .333 LLL 
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TEMPLATE 5. (continued) 
Subsubcriteria: Requested Info Being Relevant 


BDSS . 344 


LAVA « 353 


RISKWATCH .344 WY 
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TEMPLATE 6. 


Criteria: User Interface 
Subcriteria: Understandability 
Subsubcriteria: Explaining Underlying Premise 


BDSS -ouS 


LAVA vste 


RISKWATCH .313 YY 





Subsubcriteria: Premise Being Comprehendible 


BDSS 303 
LAVA . 394 


RISKWATCH .303 





Subsubcriteria: Defining Terms Unambiguously 


BDSS . 333 


LAVA - 333 


RISKWATCH .333 ZAP 





29 


TEMPLATE 6. (continued) 


Subsubcriteria: Explaining Relationships Between Phases and 
Iterations 


BDSS 333 


LAVA ~333 


RISKWATCH .333 WY 





Subsubcriteria: Identifying Decision Points Clearly 


BDSS 2353 


LAVA -330 


RISKWATCH .333 WY 
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TEMPLATE 7. 


Criteria: User Interface 
Subcriteria: Support 
Subsubcriteria: Developer Providing Support for Product 
BDSS . 333 
LAVA sao 


RISKWATCH .333 





Subsubcriteria: Providing Technical Support by Phone 


BDSS .O91 
LAVA .091 


RISKWATCH .818 





Subsubcriteria: Providing Written Documentation 


BDSS 7oo5 


LAVA 2559 


RISKHATCH .333 WY 
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TEMPLATE 7. (continued) 
Subsubcriteria: Providing On Site Training 


BDSS - 250 


LAVA . 500 


RISKHATCH .250 YWYYY 
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TEMPLATE 8. 


Criteria: Adaptability 
Subcriteria: Portability 
Subsubcriteria: Applying Across System Configurations 


BDSS .O91 


LAVA .091 


RISKWATCH .818 VY 





Subsubcriteria: Applying Across Processing Methods 


BDSS 335 


LAVA 333 


RISKWATCH .333 Yj) 





Subsubcriteria: Applying Across Different Environments 


BDSS 35 


LAVA 333 


RISKWATCH .333 LLL 
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TEMPLATE 8. (continued) 


Subsubcriteria: Applying Across All Phases of System Life 
Cycle 


BDSS 4 


LAVA .163 


RISKWATCH .540 WY 





TEMPLATE 9. 


Criteria: Adaptability 
Subcriteria: Modifiability 
Subsubcriteria: Retaining Inputs in Original Form 


BDSS - 540 


LAVA -163 


RISKWATCH .297 YY 





Subsubcriteria: Segmenting Calculations by Identifiable 
Partitions 


BDSS woo 


LAVA 5 


RISKWATCH .333 LZ 





Subsubcriteria: Modifying Software Package 


BDSS ZOO 


LAVA .250 


RISKWATCH .500 LZ. 
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TEMPLATE 10. 


Criteria: Feasibility 
Subcriteria: Availability 
Subsubcriteria: Requiring Expert Opinion for Methods Internal 
to the Organization 


BDSS 35s 


LAVA . 335 


RISKWATCH .333 WYYYYY 





Subsubcriteria: Required Data Being Internal to the 
Organization 


BDSS 1335 


LAVA oS 


RISKWATCH .333 





Subsubcriteria: Collection of Data Being Convenient at the 
Scope Desired 

BDSS .293 

LAVA 155 : 


RISKWATCH .552 
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TEMPLATE 11. 


Criteria: Feasibility 
Subcriteria: Practicality 
Subsubcriteria: Allowing Input in a Variety of Forms 


BDSS . 333 


LAVA - 330 





Subsubcriteria: Performing the Process by Available Staff 


BDSS ooo 


LAVA - 333 


RISKWATCH .333 LLL. 





Subsubcriteria: Time Being Available to Perform the Process 


BDSS 53335 


LAVA .333 pak othe - 


RISKWATCH .333 WY 
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TEMPLATE 11. (continued) 


Subsubcriteria: Obtaining Precision Economically 


BDSS Bg 6 
LAVA 3355 


RISKWATCH .333 
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TEMPLATE 12. 


Criteria: Feasibility 
Subcriteria: Scope 
Subsubcriteria: User Selecting Amount of Detail 
BDSS .053 
LAVA . 474 


RISKWATCH .474 





Subsubcriteria: Bounding Detail at the Level Desired 


BDSS . 474 
LAVA .053 


RISKWATCH .474 





Subsubcriteria: Analyzing All Data Aspects of the System 


BDSS . 333 


LAVA 530 


RISKWATCH .333 
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TEMPLATE 12. (continued) 


Subsubcriteria: Analyzing Procedural Aspects of the System 


BDSS Fee 
LAVA es io) S: 


RISKWATCH .333 





Subsubcriteria: Analyzing Personnel Aspects of the System 


BDSS .333 


LAVA Bes: 


RISKWATCH .333 ZZ 





Subsubcriteria: Analyzing Communication Aspects of the System 





BDSS .474 
LAVA ~ 474 


RISKWATCH .053 
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TEMPLATE 12. (continued) 
Subsubcriteria: Analyzing Environment of the System 


BDSS -330 


LAVA #335 


RISKWATCH .333 LZ 





dad 


TEMPLATE 13. 


Criteria: Completeness 
Subcriteria: Scope 
Subsubcriteria: User Selecting Amount of Detail 
BDSS .053 
LAVA .474 


RISKWATCH .474 





Subsubcriteria: Bounding Detail at the Level Desired 


BDSS 474 
LAVA .053 


RISKWATCH .474 





Subsubcriteria: Analyzing All Data Aspects of the System 


BDSS . 333 


LAVA 6333 


RISKWATCH .333 LZ 
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TEMPLATE 13. (continued) 
Subsubcriteria: Analyzing Procedural Aspects of the System 


BDSS 7o90 


LAVA eC) 


RISKWATCH .333 LLL 





Subsubcriteria: Analyzing Personnel Aspects of the System 


BDSS Fe 


LAVA SERS) 


RISKWATCH .333 LIZA: 





Subsubcriteria: Analyzing Communication Aspects of the System 


BDSS .474 
LAVA -474 


RISKWATCH .053 
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TEMPLATE 13. (continued) 


Subsubcriteria: Analyzing Environment of the System 


BDSS 333 
LAVA 339 


RISKWATCH .333 
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TEMPLATE 14. 


Criteria: Completeness 

Subcriteria: Elements 
Subsubcriteria: Comprehensively Considering Assets 
BDSS .540 
LAVA . 297 


RISKWATCH .163 





Subsubcriteria: Comprehensively Considering Threat Agents 


BDSS 2-333 
LAVA . 333 


RISKWATCH .333 





Subsubcriteria: Comprehensively Considering Threat Events 


BDSS .333 


LAVA .333 


RISKWATCH .333 WY 
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TEMPLATE 14. (continued) 


Subsubcriteria: Comprehensively Considering Safeguards 


BDSS - 540 
LAVA . 163 


RISKWATCH .297 





Subsubcriteria: Comprehensively Considering Vulnerabilities 


BDSS . 400 
LAVA - 400 


RISKWATCH .200 





Subsubcriteria: Considering Outcomes 


BDSS . 500 


LAVA . 250 


RISKWATCH .250 Yj 
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TEMPLATE 15. 


Criteria: Completeness 
Subcriteria: Attributes 


Subsubcriteria: Considering Asset Values 


BDSS - 361 
LAVA .278 


RISKWATCH .361 





Subsubcriteria: Considering Potency of Threat Agents 


BDSS 5 SILLS 
LAVA 344 


RISKWATCH .344 


Subsubcriteria: 
BDSS 2339 
LAVA .333 


RISKWATCH .333 





Considering Undesirability of Threat Events 


YA: 
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TEMPLATE 15. (continued) 


Subsubcriteria: Considering Effectiveness of Safeguards 


BDSS - 344 
LAVA Ae 8 


RISKWATCH .344 





Subsubcriteria: Considering Severity of Outcomes 


BDSS Jus 
LAVA Beso) e 


RISKWATCH .333 





Subsubcriteria: Considering Probabilities of the Occurrence 
of Threat Events 


BDSS - 400 


LAVA 200 


RISKWATCH .400 WY” 
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TEMPLATE 16. 


Criteria: Validity 


Subcriteria: Relevancy 
Subsubcriteria: 


Expressing Results in Terms of Solutions 
Rather than Specifics 


BDSS 330 


LAVA FOS 


RISKWATCH .333 LLLP 





Subsubcriteria: Results Relating to Significant Areas of Need 
BDSS 353 
LAVA .294 


RISKWATCH .353 





Subsubcriteria: Results Fulfilling Mandated Requirements and 
Regulations 

BDSS .297 

LAVA .163 


RISKWATCH .540 WQWY},»-/VyV0—V/V4—Vy 
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TEMPLATE 16. (continued) 


Subsubcriteria: Output Results Being Qualitative — 





BDSS .090 
LAVA 820 


RISKWATCH .090 








Subsubcriteria: Output Results Being Quantitative 


BDSS ee 
LAVA “S38 


RISKWATCH .333 
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TEMPLATE 17. 


Criteria: Validity 

Subcriteria: Scope 
Subsubcriteria: User Selecting Amount of Detail 
BDSS .053 
LAVA 474 


RISKWATCH .474 





Subsubcriteria: Bounding Detail at the Level Desired 


BDSS - 474 


LAVA .053 


RISKWATCH .474 LY 





Subsubcriteria: Analyzing All Data Aspects of the System 


BDSS wo S 


LAVA soO8 


RISKWATCH .333 LLL 





a2 1 


TEMPLATE 17. (continued) 
Subsubcriteria: Analyzing Procedural Aspects of the System 


BDSS 7303 


LAVA .333 


RISKWATCH .333 LLL 





Subsubcriteria: Analyzing Personnel Aspects of the System 


BDSS Too 3 


LAVA 5 


RISKWATCH .333 LY 





Subsubcriteria: Analyzing Communication Aspects of the System 


BDSS . 474 
LAVA 474 


RISKWATCH .053 





122 


TEMPLATE 17. (continued) 
Subsubcriteria: Analyzing Environment of the System 


BDSS ae 


LAVA «333 


RISKWATCH .333 LDA 
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TEMPLATE 18. 


Criteria: Validity 
Subcriteria: Practicality 
Subsubcriteria: Allowing Input in a Variety of Forms 
BDSS +333 
LAVA ooo5 


RISKWATCH .333 





Subsubcriteria: Performing the Process by Available Staff 


BDSS Fee) 4 
LAVA . 303 


RISKWATCH .365 





Subsubcriteria: Time Being Available to Perform the Process 


BDSS .344 


LAVA 13a 


RISKWATCH .344 LZ 
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TEMPLATE 18. (continued) 
Subsubcriteria: Obtaining Precision Economically 


BDSS S50 


LAVA Be S 


RISKWATCH .333 WY 





Subsubcriteria: Analyzing Personnel Aspects 


BDSS x55 5 
LAVA oS 


RISKWATCH .333 





Subsubcriteria: Analyzing Communication Aspects 


BDSS -474 
LAVA -474 


RISKWATCH .053 
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TEMPLATE 18. (continued) 
Subsubcriteria: Analyzing Environment Aspects 


BDSS oo 


LAVA -3o3 


RISKWATCH .333 Wy 
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Subsubcriteria: 


TEMPLATE 19. 


Credibility 
Intuitiveness 


Criteria: 
Subcriteria: 


Delineating the Relationships Between the 
Results 








BDSS -335 
LAVA 333 s , 
RISKWATCH .333 Zj7 
Subsubcriteria: Output Being a Perceivable Relationship With 
the Inputs 
BDSS SooS 
LAVA 333 
RISKWATCH .333 tH 
£2 
Subsubcriteria: Analyzing All Data Aspects 
BDSS cc! 
LAVA 333 Be 
RISKWATCH .333 YW 





TEMPLATE 19. (continued) 


Subsubcriteria: Analyzing Procedural Aspects 


BDSS 333 
LAVA 333 


RISKWATCH .333 
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TEMPLATE 20. 


Criteria: Credibility 
Subcriteria: Reliability 
Subsubcriteria: Reducing the Introduction of Personal Bias 


BDSS yoOo° 
LAVA sooo 
RISKWATCH .333 





Subsubcriteria: Reducing the Impact of Uncertainty 


BDSS » 333 


LAVA 393 


RISKWATCH .333 WY 
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APPENDIX D. CERTS DSS RESULTS FOR WIDE AREA NETWORK CASE 
STUDY 


TEMPLATE 1. 
Distributed Wide Area Network 
A COMPARATIVE EVALUATION METHOD FOR RISK MANAGEMENT TOOLS 
Sorted Synthesis of Leaf Nodes with respect to GOAL 
OVERALL INCONSISTENCY INOEX = 0.00 
RISKWTCH 0.365 xe 
LAVA eo SSS SSS eee 
BDSS C—O SE ESS SSS 
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TEMPLATE 2. 


CRITERIA ALTERNATIVES 
CONSISTY 
Bz} 159 


BDSS 
Ts ea (ES) de 


Laue 
ez SSO ERK 


ADAPTITY RISKWTCH 
Ese 240 GC eee ieeey 


FEASBITY 
O 5 


USEI FACE 
ae 35: 


COMPLETE 
Al 031 


VALIDITY 
Sa: 


CREDIBTY 
EES) 1e 
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TEMPLATE 3. 


4 PERPOBRARCE WITH RESPECT 10 GOAL FOB WODES BELOW: GOAL 


USEIFACE sa “yt ei 


CORSISTY | ADPTITY COMPLETE REDIBTY 
| Patan | 





—— HS oe TAR IT 


TEMPLATE 4. 


CRITERIA ALTERNATIVES 


CONSISTY 
VHAART It VLE eo IPE) ddd 


USEIFACE = 
ee | SRT 


ADAPTITY RISKWTCH 
Exess 22 Wh eed Ped 196 


FEASBITY 
DH 


COMPLETE 
A 029 


VALIDITY 
FA 463 





CREDIBTY 
ES .@6 
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APPENDIX E. CERTS DSS RESULTS FOR BIOMED CASE STUDY 


TEMPLATE 1. 


System Under Development - Biomed 
A COMPARATIVE EVALUATION METHOD FOR RISK MANAGEMENT TOOLS 


Sorted Synthesis of Leaf Nodes with respect to GOAL 


OVERALL INCONSISTENCY INDEX = 0.00 
BDSS — i _—E—EL_—SEEe SSS SSS eee 
LAVA 00 SSS —_—_——— SS L———————E 


RISKWTCH 0.309 
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TEMPLATE 2. 


CRITERIA ALTERNATIVES 
CONSISTY 
eee] 159 


LISEI FACE 
Ba 


ADAPTITY 
05 


FEASBITY 
031 


ey E 
J .da8 


VALIDITY 
Ee) 1d 


CREDIBTY 
Eee) ado 
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TEMPLATE 3. 


‘ PERFORMANCE WITH RESPECT 10 GOAL FOR MODES BELOW: GOAL 


| 

a 

& 

I 
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"y ‘) oo a 





é 
’ 
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a ad COMPLETE 
| : re 


—— BSS TAA —— BIST 


TEMPLATE 4. 


CRITERIA ALTERNATIVES 


CONS ISTY 
BA Ale 


LISEIFACE 
Bo 


FEASBITY 
) 2 


COMPLETE 
RNOQe/ 


VALIDITY 
Pa) .073 


CREDIBTY 
ee] 170 





APPENDIX F. CERTS DSS RESULTS FOR DATA CENTER CASE STUDY 


TEMPLATE 1. 


Data Center 
A COMPARATIVE EVALUATION METHOD FOR RISK MANAGEMENT TOOLS 


Sorted Synthesis of Leaf Nodes with respect to GOAL 


OVERALL INCONSISTENCY INDEX = 0.00 


—_—e_n ow oe oe 
= = oe oe oe 
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TEMPLATE 2. 


CRITERIA ALTERNATIVES 
CONSISTY BDSS 
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LISEIFACE 
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Bw 
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[C] it 
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VALIDITY 
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TEMPLATE 3. 


Alt,  —-PERFORNANCE WITH RESPECT 10 GOAL FOR KODES BELOW: GOAL 
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1 wt 
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